요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0+
=================================================================
Linux Base Driver for Intel(R) Ethernet Adaptive Virtual Function
=================================================================
Intel Ethernet Adaptive Virtual Function Linux driver.
Copyright(c) 2013-2018 Intel Corporation.
Contents
========
- Overview
- Identifying Your Adapter
- Additional Configurations
- Known Issues/Troubleshooting
- Support
Overview
========
This file describes the iavf Linux Base Driver. This driver was formerly
called i40evf.
The iavf driver supports the below mentioned virtual function devices and
can only be activated on kernels running the i40e or newer Physical Function
(PF) driver compiled with CONFIG_PCI_IOV. The iavf driver requires
CONFIG_PCI_MSI to be enabled.
The guest OS loading the iavf driver must support MSI-X interrupts.
Identifying Your Adapter
========================
The driver in this kernel is compatible with devices based on the following:
* Intel(R) XL710 X710 Virtual Function
* Intel(R) X722 Virtual Function
* Intel(R) XXV710 Virtual Function
* Intel(R) Ethernet Adaptive Virtual Function
For the best performance, make sure the latest NVM/FW is installed on your
device.
For information on how to identify your adapter, and for the latest NVM/FW
images and Intel network drivers, refer to the Intel Support website:
https://www.intel.com/support
Additional Features and Configurations
======================================
Viewing Link Messages
---------------------
Link messages will not be displayed to the console if the distribution is
restricting system messages. In order to see network driver link messages on
your console, set dmesg to eight by entering the following::
# dmesg -n 8
NOTE:
This setting is not saved across reboots.
ethtool
-------
The driver utilizes the ethtool interface for driver configuration and
diagnostics, as well as displaying statistical information. The latest ethtool
version is required for this functionality. Download it at:
https://www.kernel.org/pub/software/network/ethtool/
Setting VLAN Tag Stripping
--------------------------
If you have applications that require Virtual Functions (VFs) to receive
packets with VLAN tags, you can disable VLAN tag stripping for the VF. The
Physical Function (PF) processes requests issued from the VF to enable or
disable VLAN tag stripping. Note that if the PF has assigned a VLAN to a VF,
then requests from that VF to set VLAN tag stripping will be ignored.
To enable/disable VLAN tag stripping for a VF, issue the following command
from inside the VM in which you are running the VF::
# ethtool -K <if_name> rxvlan on/off
or alternatively::
# ethtool --offload <if_name> rxvlan on/off
Adaptive Virtual Function
-------------------------
Adaptive Virtual Function (AVF) allows the virtual function driver, or VF, to
adapt to changing feature sets of the physical function driver (PF) with which
it is associated. This allows system administrators to update a PF without
having to update all the VFs associated with it. All AVFs have a single common
device ID and branding string.
AVFs have a minimum set of features known as "base mode," but may provide
additional features depending on what features are available in the PF with
which the AVF is associated. The following are base mode features:
- 4 Queue Pairs (QP) and associated Configuration Status Registers (CSRs)
for Tx/Rx
- i40e descriptors and ring format
- Descriptor write-back completion
- 1 control queue, with i40e descriptors, CSRs and ring format
- 5 MSI-X interrupt vectors and corresponding i40e CSRs
- 1 Interrupt Throttle Rate (ITR) index
- 1 Virtual Station Interface (VSI) per VF
- 1 Traffic Class (TC), TC0
- Receive Side Scaling (RSS) with 64 entry indirection table and key,
configured through the PF
- 1 unicast MAC address reserved per VF
- 16 MAC address filters for each VF
- Stateless offloads - non-tunneled checksums
- AVF device ID
- HW mailbox is used for VF to PF communications (including on Windows)
IEEE 802.1ad (QinQ) Support
---------------------------
The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN
IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as
"tags," and multiple VLAN IDs are thus referred to as a "tag stack." Tag stacks
allow L2 tunneling and the ability to segregate traffic within a particular
VLAN ID, among other uses.
The following are examples of how to configure 802.1ad (QinQ)::
# ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
# ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371
Where "24" and "371" are example VLAN IDs.
NOTES:
Receive checksum offloads, cloud filters, and VLAN acceleration are not
supported for 802.1ad (QinQ) packets.
Application Device Queues (ADq)
-------------------------------
Application Device Queues (ADq) allows you to dedicate one or more queues to a
specific application. This can reduce latency for the specified application,
and allow Tx traffic to be rate limited per application. Follow the steps below
to set ADq.
Requirements:
- The sch_mqprio, act_mirred and cls_flower modules must be loaded
- The latest version of iproute2
- If another driver (for example, DPDK) has set cloud filters, you cannot
enable ADQ
- Depending on the underlying PF device, ADQ cannot be enabled when the
following features are enabled:
+ Data Center Bridging (DCB)
+ Multiple Functions per Port (MFP)
+ Sideband Filters
1. Create traffic classes (TCs). Maximum of 8 TCs can be created per interface.
The shaper bw_rlimit parameter is optional.
Example: Sets up two tcs, tc0 and tc1, with 16 queues each and max tx rate set
to 1Gbit for tc0 and 3Gbit for tc1.
::
tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1
queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit
max_rate 1Gbit 3Gbit
map: priority mapping for up to 16 priorities to tcs (e.g. map 0 0 0 0 1 1 1 1
sets priorities 0-3 to use tc0 and 4-7 to use tc1)
queues: for each tc, <num queues>@<offset> (e.g. queues 16@0 16@16 assigns
16 queues to tc0 at offset 0 and 16 queues to tc1 at offset 16. Max total
number of queues for all tcs is 64 or number of cores, whichever is lower.)
hw 1 mode channel: ‘channel’ with ‘hw’ set to 1 is a new new hardware
offload mode in mqprio that makes full use of the mqprio options, the
TCs, the queue configurations, and the QoS parameters.
shaper bw_rlimit: for each tc, sets minimum and maximum bandwidth rates.
Totals must be equal or less than port speed.
For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network
monitoring tools such as ``ifstat`` or ``sar -n DEV [interval] [number of samples]``
NOTE:
Setting up channels via ethtool (ethtool -L) is not supported when the
TCs are configured using mqprio.
2. Enable HW TC offload on interface::
# ethtool -K <interface> hw-tc-offload on
3. Apply TCs to ingress (RX) flow of interface::
# tc qdisc add dev <interface> ingress
NOTES:
- Run all tc commands from the iproute2 <pathtoiproute2>/tc/ directory
- ADq is not compatible with cloud filters
- Setting up channels via ethtool (ethtool -L) is not supported when the TCs
are configured using mqprio
- You must have iproute2 latest version
- NVM version 6.01 or later is required
- ADq cannot be enabled when any the following features are enabled: Data
Center Bridging (DCB), Multiple Functions per Port (MFP), or Sideband Filters
- If another driver (for example, DPDK) has set cloud filters, you cannot
enable ADq
- Tunnel filters are not supported in ADq. If encapsulated packets do arrive
in non-tunnel mode, filtering will be done on the inner headers. For example,
for VXLAN traffic in non-tunnel mode, PCTYPE is identified as a VXLAN
encapsulated packet, outer headers are ignored. Therefore, inner headers are
matched.
- If a TC filter on a PF matches traffic over a VF (on the PF), that traffic
will be routed to the appropriate queue of the PF, and will not be passed on
the VF. Such traffic will end up getting dropped higher up in the TCP/IP
stack as it does not match PF address data.
- If traffic matches multiple TC filters that point to different TCs, that
traffic will be duplicated and sent to all matching TC queues. The hardware
switch mirrors the packet to a VSI list when multiple filters are matched.
Known Issues/Troubleshooting
============================
Bonding fails with VFs bound to an Intel(R) Ethernet Controller 700 series device
---------------------------------------------------------------------------------
If you bind Virtual Functions (VFs) to an Intel(R) Ethernet Controller 700
series based device, the VF slaves may fail when they become the active slave.
If the MAC address of the VF is set by the PF (Physical Function) of the
device, when you add a slave, or change the active-backup slave, Linux bonding
tries to sync the backup slave's MAC address to the same MAC address as the
active slave. Linux bonding will fail at this point. This issue will not occur
if the VF's MAC address is not set by the PF.
Traffic Is Not Being Passed Between VM and Client
-------------------------------------------------
You may not be able to pass traffic between a client system and a
Virtual Machine (VM) running on a separate host if the Virtual Function
(VF, or Virtual NIC) is not in trusted mode and spoof checking is enabled
on the VF. Note that this situation can occur in any combination of client,
host, and guest operating system. For information on how to set the VF to
trusted mode, refer to the section "VLAN Tag Packet Steering" in this
readme document. For information on setting spoof checking, refer to the
section "MAC and VLAN anti-spoofing feature" in this readme document.
Do not unload port driver if VF with active VM is bound to it
-------------------------------------------------------------
Do not unload a port's driver if a Virtual Function (VF) with an active Virtual
Machine (VM) is bound to it. Doing so will cause the port to appear to hang.
Once the VM shuts down, or otherwise releases the VF, the command will complete.
Using four traffic classes fails
--------------------------------
Do not try to reserve more than three traffic classes in the iavf driver. Doing
so will fail to set any traffic classes and will cause the driver to write
errors to stdout. Use a maximum of three queues to avoid this issue.
Multiple log error messages on iavf driver removal
--------------------------------------------------
If you have several VFs and you remove the iavf driver, several instances of
the following log errors are written to the log::
Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err ok
Unable to send the message to VF 2 aq_err 12
ARQ Overflow Error detected
Virtual machine does not get link
---------------------------------
If the virtual machine has more than one virtual port assigned to it, and those
virtual ports are bound to different physical ports, you may not get link on
all of the virtual ports. The following command may work around the issue::
# ethtool -r <PF>
Where <PF> is the PF interface in the host, for example: p5p1. You may need to
run the command more than once to get link on all virtual ports.
MAC address of Virtual Function changes unexpectedly
----------------------------------------------------
If a Virtual Function's MAC address is not assigned in the host, then the VF
(virtual function) driver will use a random MAC address. This random MAC
address may change each time the VF driver is reloaded. You can assign a static
MAC address in the host machine. This static MAC address will survive
a VF driver reload.
Driver Buffer Overflow Fix
--------------------------
The fix to resolve CVE-2016-8105, referenced in Intel SA-00069
https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00069.html
is included in this and future versions of the driver.
Multiple Interfaces on Same Ethernet Broadcast Network
------------------------------------------------------
Due to the default ARP behavior on Linux, it is not possible to have one system
on two IP networks in the same Ethernet broadcast domain (non-partitioned
switch) behave as expected. All Ethernet interfaces will respond to IP traffic
for any IP address assigned to the system. This results in unbalanced receive
traffic.
If you have multiple interfaces in a server, either turn on ARP filtering by
entering::
# echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
NOTE:
This setting is not saved across reboots. The configuration change can be
made permanent by adding the following line to the file /etc/sysctl.conf::
net.ipv4.conf.all.arp_filter = 1
Another alternative is to install the interfaces in separate broadcast domains
(either in different switches or in a switch partitioned to VLANs).
Rx Page Allocation Errors
-------------------------
'Page allocation failure. order:0' errors may occur under stress.
This is caused by the way the Linux kernel reports this stressed condition.
Support
=======
For general information, go to the Intel support website at:
https://support.intel.com
If an issue is identified with the released source code on the supported kernel
with a supported adapter, email the specific information related to the issue
to [email protected].
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
개요와 adapter 식별
1-48이 문서는 `GPL-2.0+` 라이선스를 따릅니다.
Intel(R) Ethernet Adaptive Virtual Function용 Linux Base Driver
Intel Ethernet Adaptive Virtual Function Linux driver.
Copyright(c) 2013-2018 Intel Corporation.
목차
- 개요
- Adapter 식별
- 추가 configuration
- 알려진 문제와 troubleshooting
- 지원
개요
이 문서는 iavf Linux Base Driver를 설명합니다. 이 driver의 이전 이름은 i40evf였습니다.
iavf driver는 아래에 열거한 virtual function device를 지원합니다. `CONFIG_PCI_IOV`를 사용해 compile한 i40e 또는 그보다 새로운 Physical Function(PF) driver가 실행 중인 kernel에서만 활성화할 수 있습니다. iavf driver를 사용하려면 `CONFIG_PCI_MSI`도 활성화해야 합니다.
iavf driver를 load하는 guest OS는 MSI-X interrupt를 지원해야 합니다.
Adapter 식별
이 kernel의 driver는 다음 device 기반 adapter와 호환됩니다.
- Intel(R) XL710 X710 Virtual Function
- Intel(R) X722 Virtual Function
- Intel(R) XXV710 Virtual Function
- Intel(R) Ethernet Adaptive Virtual Function
최상의 performance를 위해 device에 최신 NVM/FW를 설치하십시오.
adapter 식별 방법, 최신 NVM/FW image와 Intel network driver 정보는 Intel Support website `https://www.intel.com/support`를 참고하십시오.
.. SPDX-License-Identifier: GPL-2.0+
=================================================================
Linux Base Driver for Intel(R) Ethernet Adaptive Virtual Function
=================================================================
Intel Ethernet Adaptive Virtual Function Linux driver.
Copyright(c) 2013-2018 Intel Corporation.
Contents
========
- Overview
- Identifying Your Adapter
- Additional Configurations
- Known Issues/Troubleshooting
- Support
Overview
========
This file describes the iavf Linux Base Driver. This driver was formerly
called i40evf.
The iavf driver supports the below mentioned virtual function devices and
can only be activated on kernels running the i40e or newer Physical Function
(PF) driver compiled with CONFIG_PCI_IOV. The iavf driver requires
CONFIG_PCI_MSI to be enabled.
The guest OS loading the iavf driver must support MSI-X interrupts.
Identifying Your Adapter
========================
The driver in this kernel is compatible with devices based on the following:
* Intel(R) XL710 X710 Virtual Function
* Intel(R) X722 Virtual Function
* Intel(R) XXV710 Virtual Function
* Intel(R) Ethernet Adaptive Virtual Function
For the best performance, make sure the latest NVM/FW is installed on your
device.
For information on how to identify your adapter, and for the latest NVM/FW
images and Intel network drivers, refer to the Intel Support website:
https://www.intel.com/support
Link message, ethtool과 VLAN tag stripping
49-86추가 기능과 configuration
Link message 보기
distribution이 system message를 제한하면 link message가 console에 표시되지 않습니다. console에서 network driver link message를 보려면 dmesg level을 8로 설정합니다.
# dmesg -n 8
참고: 이 설정은 reboot 후 유지되지 않습니다.
ethtool
driver는 configuration과 진단, 통계 표시를 위해 ethtool interface를 사용합니다. 이 기능에는 최신 ethtool version이 필요하며 `https://www.kernel.org/pub/software/network/ethtool/`에서 받을 수 있습니다.
VLAN tag stripping 설정
Virtual Function(VF)이 VLAN tag가 붙은 packet을 받아야 하는 application에서는 VF의 VLAN tag stripping을 비활성화할 수 있습니다. Physical Function(PF)은 VLAN tag stripping을 켜거나 끄기 위한 VF의 요청을 처리합니다.
PF가 VF에 VLAN을 이미 할당했다면 해당 VF가 보낸 VLAN tag stripping 변경 요청은 무시됩니다.
VF를 실행하는 VM 내부에서 다음 command 중 하나로 VLAN tag stripping을 켜거나 끕니다.
# ethtool -K <if_name> rxvlan on/off
# ethtool --offload <if_name> rxvlan on/off
Additional Features and Configurations
======================================
Viewing Link Messages
---------------------
Link messages will not be displayed to the console if the distribution is
restricting system messages. In order to see network driver link messages on
your console, set dmesg to eight by entering the following::
# dmesg -n 8
NOTE:
This setting is not saved across reboots.
ethtool
-------
The driver utilizes the ethtool interface for driver configuration and
diagnostics, as well as displaying statistical information. The latest ethtool
version is required for this functionality. Download it at:
https://www.kernel.org/pub/software/network/ethtool/
Setting VLAN Tag Stripping
--------------------------
If you have applications that require Virtual Functions (VFs) to receive
packets with VLAN tags, you can disable VLAN tag stripping for the VF. The
Physical Function (PF) processes requests issued from the VF to enable or
disable VLAN tag stripping. Note that if the PF has assigned a VLAN to a VF,
then requests from that VF to set VLAN tag stripping will be ignored.
To enable/disable VLAN tag stripping for a VF, issue the following command
from inside the VM in which you are running the VF::
# ethtool -K <if_name> rxvlan on/off
or alternatively::
# ethtool --offload <if_name> rxvlan on/off
Adaptive Virtual Function과 QinQ
87-134Adaptive Virtual Function
Adaptive Virtual Function(AVF)은 virtual function driver, 즉 VF가 자신과 연결된 Physical Function driver(PF)의 변경되는 feature set에 적응하게 합니다. 따라서 system administrator는 연결된 모든 VF를 함께 update하지 않고도 PF를 update할 수 있습니다. 모든 AVF는 하나의 공통 device ID와 branding string을 사용합니다.
AVF에는 `base mode`라는 최소 feature set이 있으며, 연결된 PF에서 제공하는 기능에 따라 추가 feature를 제공할 수 있습니다. base mode 기능은 다음과 같습니다.
- Tx/Rx용 4개 Queue Pair(QP)와 관련 Configuration Status Register(CSR)
- i40e descriptor와 ring format
- descriptor write-back completion
- i40e descriptor, CSR, ring format을 사용하는 control queue 1개
- MSI-X interrupt vector 5개와 대응하는 i40e CSR
- Interrupt Throttle Rate(ITR) index 1개
- VF마다 Virtual Station Interface(VSI) 1개
- Traffic Class(TC) 1개인 TC0
- PF를 통해 구성하는 64-entry indirection table과 key 기반 Receive Side Scaling(RSS)
- VF마다 예약된 unicast MAC address 1개
- VF마다 MAC address filter 16개
- non-tunneled checksum용 stateless offload
- AVF device ID
- Windows를 포함한 VF-PF 통신용 hardware mailbox
IEEE 802.1ad(QinQ) 지원
QinQ로도 불리는 IEEE 802.1ad 표준은 하나의 Ethernet frame 안에 여러 VLAN ID를 넣을 수 있게 합니다. VLAN ID는 흔히 tag라고 하며, 여러 VLAN ID의 집합은 tag stack이라고 합니다. tag stack은 L2 tunneling과 특정 VLAN ID 내부의 traffic 분리 등에 사용합니다.
802.1ad QinQ 구성 예시는 다음과 같습니다.
# ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
# ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371
여기서 `24`와 `371`은 VLAN ID 예시입니다.
참고: 802.1ad(QinQ) packet에는 receive checksum offload, cloud filter와 VLAN acceleration을 지원하지 않습니다.
Adaptive Virtual Function
-------------------------
Adaptive Virtual Function (AVF) allows the virtual function driver, or VF, to
adapt to changing feature sets of the physical function driver (PF) with which
it is associated. This allows system administrators to update a PF without
having to update all the VFs associated with it. All AVFs have a single common
device ID and branding string.
AVFs have a minimum set of features known as "base mode," but may provide
additional features depending on what features are available in the PF with
which the AVF is associated. The following are base mode features:
- 4 Queue Pairs (QP) and associated Configuration Status Registers (CSRs)
for Tx/Rx
- i40e descriptors and ring format
- Descriptor write-back completion
- 1 control queue, with i40e descriptors, CSRs and ring format
- 5 MSI-X interrupt vectors and corresponding i40e CSRs
- 1 Interrupt Throttle Rate (ITR) index
- 1 Virtual Station Interface (VSI) per VF
- 1 Traffic Class (TC), TC0
- Receive Side Scaling (RSS) with 64 entry indirection table and key,
configured through the PF
- 1 unicast MAC address reserved per VF
- 16 MAC address filters for each VF
- Stateless offloads - non-tunneled checksums
- AVF device ID
- HW mailbox is used for VF to PF communications (including on Windows)
IEEE 802.1ad (QinQ) Support
---------------------------
The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN
IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as
"tags," and multiple VLAN IDs are thus referred to as a "tag stack." Tag stacks
allow L2 tunneling and the ability to segregate traffic within a particular
VLAN ID, among other uses.
The following are examples of how to configure 802.1ad (QinQ)::
# ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
# ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371
Where "24" and "371" are example VLAN IDs.
NOTES:
Receive checksum offloads, cloud filters, and VLAN acceleration are not
supported for 802.1ad (QinQ) packets.
Application Device Queues 요구사항과 TC 생성
135-186Application Device Queues(ADq)
ADq를 사용하면 하나 이상의 queue를 특정 application 전용으로 지정할 수 있습니다. 지정한 application의 latency를 줄이고 application별 Tx traffic rate limit을 적용할 수 있습니다.
요구사항
- `sch_mqprio`, `act_mirred`, `cls_flower` module을 load해야 합니다.
- 최신 version의 iproute2가 필요합니다.
- DPDK 같은 다른 driver가 cloud filter를 설정했다면 ADq를 활성화할 수 없습니다.
- 기반 PF device에 따라 DCB, MFP 또는 Sideband Filter가 활성화된 상태에서는 ADq를 활성화할 수 없습니다.
1. Traffic Class(TC)를 만듭니다. interface마다 최대 8개 TC를 만들 수 있으며 `shaper bw_rlimit` parameter는 선택 사항입니다.
다음 예시는 TC0와 TC1에 각각 queue 16개를 배치하고 최대 Tx rate를 TC0는 1 Gbit, TC1은 3 Gbit로 설정합니다.
tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1
queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit
max_rate 1Gbit 3Gbit
`map`은 최대 16개 priority를 TC에 대응시킵니다. 예를 들어 `map 0 0 0 0 1 1 1 1`은 priority 0-3을 TC0에, 4-7을 TC1에 연결합니다.
`queues`는 각 TC에 `<queue 수>@<offset>` 형식으로 queue를 배치합니다. `16@0 16@16`은 offset 0부터 16개를 TC0에, offset 16부터 16개를 TC1에 할당합니다. 모든 TC의 전체 queue 수 상한은 64와 core 수 중 더 작은 값입니다.
`hw 1 mode channel`은 mqprio option, TC, queue configuration과 QoS parameter를 온전히 사용하는 hardware offload mode입니다.
`shaper bw_rlimit`은 TC마다 최소·최대 bandwidth rate를 설정합니다. 전체 합계는 port speed 이하여야 합니다.
`min_rate 1Gbit 3Gbit` 같은 bandwidth limit은 `ifstat` 또는 `sar -n DEV [interval] [number of samples]` 같은 network monitoring tool로 확인합니다.
참고: mqprio로 TC를 구성한 상태에서는 `ethtool -L`로 channel을 설정할 수 없습니다.
Application Device Queues (ADq)
-------------------------------
Application Device Queues (ADq) allows you to dedicate one or more queues to a
specific application. This can reduce latency for the specified application,
and allow Tx traffic to be rate limited per application. Follow the steps below
to set ADq.
Requirements:
- The sch_mqprio, act_mirred and cls_flower modules must be loaded
- The latest version of iproute2
- If another driver (for example, DPDK) has set cloud filters, you cannot
enable ADQ
- Depending on the underlying PF device, ADQ cannot be enabled when the
following features are enabled:
+ Data Center Bridging (DCB)
+ Multiple Functions per Port (MFP)
+ Sideband Filters
1. Create traffic classes (TCs). Maximum of 8 TCs can be created per interface.
The shaper bw_rlimit parameter is optional.
Example: Sets up two tcs, tc0 and tc1, with 16 queues each and max tx rate set
to 1Gbit for tc0 and 3Gbit for tc1.
::
tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1
queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit
max_rate 1Gbit 3Gbit
map: priority mapping for up to 16 priorities to tcs (e.g. map 0 0 0 0 1 1 1 1
sets priorities 0-3 to use tc0 and 4-7 to use tc1)
queues: for each tc, <num queues>@<offset> (e.g. queues 16@0 16@16 assigns
16 queues to tc0 at offset 0 and 16 queues to tc1 at offset 16. Max total
number of queues for all tcs is 64 or number of cores, whichever is lower.)
hw 1 mode channel: ‘channel’ with ‘hw’ set to 1 is a new new hardware
offload mode in mqprio that makes full use of the mqprio options, the
TCs, the queue configurations, and the QoS parameters.
shaper bw_rlimit: for each tc, sets minimum and maximum bandwidth rates.
Totals must be equal or less than port speed.
For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network
monitoring tools such as ``ifstat`` or ``sar -n DEV [interval] [number of samples]``
NOTE:
Setting up channels via ethtool (ethtool -L) is not supported when the
TCs are configured using mqprio.
ADq hardware offload와 제약
187-2202. interface에서 hardware TC offload를 활성화합니다.
# ethtool -K <interface> hw-tc-offload on
3. interface의 ingress(RX) flow에 TC를 적용합니다.
# tc qdisc add dev <interface> ingress
참고
- 모든 tc command는 iproute2의 `<pathtoiproute2>/tc/` directory에서 실행합니다.
- ADq는 cloud filter와 호환되지 않습니다.
- mqprio로 TC를 구성한 상태에서는 `ethtool -L`로 channel을 설정할 수 없습니다.
- 최신 iproute2가 필요합니다.
- NVM version 6.01 이상이 필요합니다.
- DCB, MFP 또는 Sideband Filter가 활성화되어 있으면 ADq를 활성화할 수 없습니다.
- DPDK 같은 다른 driver가 cloud filter를 설정했다면 ADq를 활성화할 수 없습니다.
- ADq는 tunnel filter를 지원하지 않습니다.
encapsulated packet이 non-tunnel mode로 들어오면 inner header를 기준으로 filtering합니다. 예를 들어 non-tunnel mode의 VXLAN traffic은 VXLAN encapsulated packet PCTYPE로 식별되고 outer header는 무시되므로 inner header가 matching 대상입니다.
PF의 TC filter가 PF를 통과하는 VF traffic과 일치하면 해당 traffic은 PF의 적절한 queue로 route되며 VF에 전달되지 않습니다. 이후 PF address data와 맞지 않아 TCP/IP stack의 상위 계층에서 drop됩니다.
traffic이 서로 다른 TC를 가리키는 여러 TC filter와 일치하면 packet은 복제되어 일치하는 모든 TC queue로 전송됩니다. 여러 filter가 일치할 때 hardware switch는 packet을 VSI list에 mirror합니다.
2. Enable HW TC offload on interface::
# ethtool -K <interface> hw-tc-offload on
3. Apply TCs to ingress (RX) flow of interface::
# tc qdisc add dev <interface> ingress
NOTES:
- Run all tc commands from the iproute2 <pathtoiproute2>/tc/ directory
- ADq is not compatible with cloud filters
- Setting up channels via ethtool (ethtool -L) is not supported when the TCs
are configured using mqprio
- You must have iproute2 latest version
- NVM version 6.01 or later is required
- ADq cannot be enabled when any the following features are enabled: Data
Center Bridging (DCB), Multiple Functions per Port (MFP), or Sideband Filters
- If another driver (for example, DPDK) has set cloud filters, you cannot
enable ADq
- Tunnel filters are not supported in ADq. If encapsulated packets do arrive
in non-tunnel mode, filtering will be done on the inner headers. For example,
for VXLAN traffic in non-tunnel mode, PCTYPE is identified as a VXLAN
encapsulated packet, outer headers are ignored. Therefore, inner headers are
matched.
- If a TC filter on a PF matches traffic over a VF (on the PF), that traffic
will be routed to the appropriate queue of the PF, and will not be passed on
the VF. Such traffic will end up getting dropped higher up in the TCP/IP
stack as it does not match PF address data.
- If traffic matches multiple TC filters that point to different TCs, that
traffic will be duplicated and sent to all matching TC queues. The hardware
switch mirrors the packet to a VSI list when multiple filters are matched.
Bonding, trusted mode와 traffic class 문제
221-255알려진 문제와 troubleshooting
Intel(R) Ethernet Controller 700 Series device에 bind된 VF에서 bonding 실패
Intel 700 Series 기반 device에 Virtual Function(VF)을 bind하면 VF slave가 active slave가 될 때 실패할 수 있습니다. PF가 VF의 MAC address를 설정한 경우 slave를 추가하거나 active-backup slave를 바꿀 때 Linux bonding이 backup slave MAC address를 active slave와 같게 동기화하려 하며 이 단계에서 실패합니다. PF가 VF MAC address를 설정하지 않았다면 이 문제는 발생하지 않습니다.
VM과 client 사이에 traffic이 전달되지 않음
Virtual Function(VF 또는 virtual NIC)이 trusted mode가 아니고 VF에서 spoof checking이 활성화되어 있으면 서로 다른 host에 있는 client system과 Virtual Machine(VM) 사이에서 traffic이 전달되지 않을 수 있습니다. client, host, guest OS의 어떤 조합에서도 발생할 수 있습니다.
VF trusted mode 설정은 이 readme의 `VLAN Tag Packet Steering` 절을, spoof checking 설정은 `MAC and VLAN anti-spoofing feature` 절을 참고하십시오.
active VM에 bind된 VF가 있으면 port driver를 unload하지 마십시오.
active VM에 bind된 VF가 있는 port의 driver를 unload하면 port가 정지한 것처럼 보입니다. VM이 종료되거나 다른 방식으로 VF를 release하면 command가 완료됩니다.
Traffic Class 4개 사용 실패
iavf driver에 3개를 초과하는 traffic class를 예약하지 마십시오. 4개 이상을 시도하면 모든 traffic class 설정이 실패하고 driver가 stdout에 error를 기록합니다. 이 문제를 피하려면 queue를 최대 3개 사용하십시오.
Known Issues/Troubleshooting
============================
Bonding fails with VFs bound to an Intel(R) Ethernet Controller 700 series device
---------------------------------------------------------------------------------
If you bind Virtual Functions (VFs) to an Intel(R) Ethernet Controller 700
series based device, the VF slaves may fail when they become the active slave.
If the MAC address of the VF is set by the PF (Physical Function) of the
device, when you add a slave, or change the active-backup slave, Linux bonding
tries to sync the backup slave's MAC address to the same MAC address as the
active slave. Linux bonding will fail at this point. This issue will not occur
if the VF's MAC address is not set by the PF.
Traffic Is Not Being Passed Between VM and Client
-------------------------------------------------
You may not be able to pass traffic between a client system and a
Virtual Machine (VM) running on a separate host if the Virtual Function
(VF, or Virtual NIC) is not in trusted mode and spoof checking is enabled
on the VF. Note that this situation can occur in any combination of client,
host, and guest operating system. For information on how to set the VF to
trusted mode, refer to the section "VLAN Tag Packet Steering" in this
readme document. For information on setting spoof checking, refer to the
section "MAC and VLAN anti-spoofing feature" in this readme document.
Do not unload port driver if VF with active VM is bound to it
-------------------------------------------------------------
Do not unload a port's driver if a Virtual Function (VF) with an active Virtual
Machine (VM) is bound to it. Doing so will cause the port to appear to hang.
Once the VM shuts down, or otherwise releases the VF, the command will complete.
Using four traffic classes fails
--------------------------------
Do not try to reserve more than three traffic classes in the iavf driver. Doing
so will fail to set any traffic classes and will cause the driver to write
errors to stdout. Use a maximum of three queues to avoid this issue.
Driver 제거, virtual link와 MAC address 문제
256-283iavf driver 제거 시 여러 log error message
VF가 여러 개 있는 상태에서 iavf driver를 제거하면 다음 log error가 여러 번 기록됩니다.
Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err ok
Unable to send the message to VF 2 aq_err 12
ARQ Overflow Error detected
Virtual machine이 link를 얻지 못함
virtual machine에 둘 이상의 virtual port가 할당되고 그 port들이 서로 다른 physical port에 bind되어 있으면 모든 virtual port에서 link를 얻지 못할 수 있습니다. 다음 command로 문제를 우회할 수 있습니다.
# ethtool -r <PF>
`<PF>`는 host의 PF interface이며 예를 들면 `p5p1`입니다. 모든 virtual port에서 link를 얻으려면 command를 여러 번 실행해야 할 수 있습니다.
Virtual Function의 MAC address가 예기치 않게 변경됨
host에서 VF MAC address를 할당하지 않으면 VF driver가 random MAC address를 사용합니다. 이 random address는 VF driver를 reload할 때마다 바뀔 수 있습니다. host machine에 static MAC address를 할당하면 VF driver reload 후에도 유지됩니다.
Multiple log error messages on iavf driver removal
--------------------------------------------------
If you have several VFs and you remove the iavf driver, several instances of
the following log errors are written to the log::
Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err ok
Unable to send the message to VF 2 aq_err 12
ARQ Overflow Error detected
Virtual machine does not get link
---------------------------------
If the virtual machine has more than one virtual port assigned to it, and those
virtual ports are bound to different physical ports, you may not get link on
all of the virtual ports. The following command may work around the issue::
# ethtool -r <PF>
Where <PF> is the PF interface in the host, for example: p5p1. You may need to
run the command more than once to get link on all virtual ports.
MAC address of Virtual Function changes unexpectedly
----------------------------------------------------
If a Virtual Function's MAC address is not assigned in the host, then the VF
(virtual function) driver will use a random MAC address. This random MAC
address may change each time the VF driver is reloaded. You can assign a static
MAC address in the host machine. This static MAC address will survive
a VF driver reload.
Security fix, ARP filtering과 page allocation
284-318Driver buffer overflow 수정
Intel SA-00069에서 참조한 CVE-2016-8105 해결 수정은 이 driver version과 이후 version에 포함되어 있습니다. advisory는 `https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00069.html`을 참고하십시오.
같은 Ethernet broadcast network의 여러 interface
Linux의 default ARP 동작 때문에 partition되지 않은 같은 Ethernet broadcast domain에서 한 system이 두 IP network에 연결될 때 기대한 대로 동작하지 않습니다. 모든 Ethernet interface가 system에 할당된 어떤 IP address로 온 traffic에도 응답하므로 receive traffic이 불균형해집니다.
server에 interface가 여러 개라면 ARP filtering을 활성화합니다.
# echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
참고: 이 설정은 reboot 후 유지되지 않습니다. `/etc/sysctl.conf`에 다음 줄을 추가하면 영구적으로 적용할 수 있습니다.
net.ipv4.conf.all.arp_filter = 1
또 다른 방법은 interface를 서로 다른 broadcast domain에 두는 것입니다. 서로 다른 switch를 사용하거나 switch를 VLAN으로 partition할 수 있습니다.
Rx page allocation error
부하가 높은 상황에서 `Page allocation failure. order:0` error가 발생할 수 있습니다. 이는 Linux kernel이 이 stress 상태를 보고하는 방식 때문에 나타납니다.
Driver Buffer Overflow Fix
--------------------------
The fix to resolve CVE-2016-8105, referenced in Intel SA-00069
https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00069.html
is included in this and future versions of the driver.
Multiple Interfaces on Same Ethernet Broadcast Network
------------------------------------------------------
Due to the default ARP behavior on Linux, it is not possible to have one system
on two IP networks in the same Ethernet broadcast domain (non-partitioned
switch) behave as expected. All Ethernet interfaces will respond to IP traffic
for any IP address assigned to the system. This results in unbalanced receive
traffic.
If you have multiple interfaces in a server, either turn on ARP filtering by
entering::
# echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
NOTE:
This setting is not saved across reboots. The configuration change can be
made permanent by adding the following line to the file /etc/sysctl.conf::
net.ipv4.conf.all.arp_filter = 1
Another alternative is to install the interfaces in separate broadcast domains
(either in different switches or in a switch partitioned to VLANs).
Rx Page Allocation Errors
-------------------------
'Page allocation failure. order:0' errors may occur under stress.
This is caused by the way the Linux kernel reports this stressed condition.
지원
319-326지원
일반 정보는 Intel support website `https://support.intel.com`을 참고하십시오.
지원되는 adapter와 kernel에서 공개 source code의 문제가 확인되면 문제에 관한 구체적인 정보를 `[email protected]`로 보내십시오.
Support
=======
For general information, go to the Intel support website at:
https://support.intel.com
If an issue is identified with the released source code on the supported kernel
with a supported adapter, email the specific information related to the issue
to [email protected].
요약·해설
iavf.rst:1-326iavf는 Intel 700 Series 계열 PF가 제공하는 기능에 맞춰 동작하는 adaptive VF driver입니다. guest의 MSI-X 지원과 PF·kernel configuration이 선행되어야 하며, 실제 기능 범위는 공통 base mode에 PF가 협상한 기능을 더해 결정됩니다. ADq, VLAN, QinQ와 bonding은 PF가 가진 전역 상태와 queue resource 제약을 함께 확인해야 합니다.
host PF부터 guest interrupt support까지 필요한 의존 관계입니다.
PF 기능과 무관하게 제공되는 최소 VF resource입니다.
VM의 ethtool 요청이 PF 정책에 따라 적용되는 과정입니다.
outer 802.1ad와 inner 802.1Q tag stack 운용 조건입니다.
application 전용 queue와 rate limit을 만드는 핵심 단계입니다.
활성화 전에 확인할 상한과 배타 기능입니다.
VF lifecycle과 network policy에서 자주 마주치는 증상입니다.