Linux Base Driver for the Intel(R) Ethernet 10 Gigabit PCI Express Adapters

개요와 adapter 식별

이 문서는 `GPL-2.0+` 라이선스를 따릅니다.

Intel(R) Ethernet 10 Gigabit PCI Express Adapter용 Linux Base Driver

Intel 10 Gigabit Linux driver.

Copyright(c) 1999-2018 Intel Corporation.

목차

• Adapter 식별
• Command line parameter
• 추가 configuration
• 알려진 문제
• 지원

Adapter 식별

이 driver는 다음 controller 기반 device와 호환됩니다.

• Intel(R) Ethernet Controller 82598
• Intel(R) Ethernet Controller 82599
• Intel(R) Ethernet Controller X520
• Intel(R) Ethernet Controller X540
• Intel(R) Ethernet Controller x550
• Intel(R) Ethernet Controller X552
• Intel(R) Ethernet Controller X553

adapter 식별 방법과 최신 Intel network driver는 Intel Support website `https://www.intel.com/support`를 참고하십시오.

.. SPDX-License-Identifier: GPL-2.0+

===========================================================================
Linux Base Driver for the Intel(R) Ethernet 10 Gigabit PCI Express Adapters
===========================================================================

Intel 10 Gigabit Linux driver.
Copyright(c) 1999-2018 Intel Corporation.

Contents
========

- Identifying Your Adapter
- Command Line Parameters
- Additional Configurations
- Known Issues
- Support

Identifying Your Adapter
========================
The driver is compatible with devices based on the following:

 * Intel(R) Ethernet Controller 82598
 * Intel(R) Ethernet Controller 82599
 * Intel(R) Ethernet Controller X520
 * Intel(R) Ethernet Controller X540
 * Intel(R) Ethernet Controller x550
 * Intel(R) Ethernet Controller X552
 * Intel(R) Ethernet Controller X553

For information on how to identify your adapter, and for the latest Intel
network drivers, refer to the Intel Support website:
https://www.intel.com/support

82599 기반 SFP+ optics

교체식 optics를 사용하는 SFP+ device

82599 기반 adapter

82599 기반 Intel(R) Network Adapter가 Intel optics와 함께 제공되었거나 Intel(R) Ethernet Server Adapter X520-2라면 Intel optics와 아래에 기재된 direct attach cable만 지원합니다.

82599 기반 SFP+ device를 back-to-back으로 연결할 때는 ethtool로 양쪽 Speed를 같게 설정해야 합니다. 서로 다른 speed 설정을 섞으면 결과가 달라질 수 있습니다.

다음은 일정 수준의 시험을 거친 third-party SFP+ module 목록입니다. 모든 module이 모든 device에 적용되는 것은 아닙니다.

82599 기반 adapter는 SFF-8431 v4.1과 SFF-8472 v10.4 규격을 준수하는 모든 passive direct attach cable과 active limiting direct attach cable을 지원합니다.

SFP+ Devices with Pluggable Optics
----------------------------------

82599-BASED ADAPTERS
~~~~~~~~~~~~~~~~~~~~
NOTES:
- If your 82599-based Intel(R) Network Adapter came with Intel optics or is an
Intel(R) Ethernet Server Adapter X520-2, then it only supports Intel optics
and/or the direct attach cables listed below.
- When 82599-based SFP+ devices are connected back to back, they should be set
to the same Speed setting via ethtool. Results may vary if you mix speed
settings.

+---------------+---------------------------------------+------------------+
| Supplier      | Type                                  | Part Numbers     |
+===============+=======================================+==================+
| SR Modules                                                               |
+---------------+---------------------------------------+------------------+
| Intel         | DUAL RATE 1G/10G SFP+ SR (bailed)     | FTLX8571D3BCV-IT |
+---------------+---------------------------------------+------------------+
| Intel         | DUAL RATE 1G/10G SFP+ SR (bailed)     | AFBR-703SDZ-IN2  |
+---------------+---------------------------------------+------------------+
| Intel         | DUAL RATE 1G/10G SFP+ SR (bailed)     | AFBR-703SDDZ-IN1 |
+---------------+---------------------------------------+------------------+
| LR Modules                                                               |
+---------------+---------------------------------------+------------------+
| Intel         | DUAL RATE 1G/10G SFP+ LR (bailed)     | FTLX1471D3BCV-IT |
+---------------+---------------------------------------+------------------+
| Intel         | DUAL RATE 1G/10G SFP+ LR (bailed)     | AFCT-701SDZ-IN2  |
+---------------+---------------------------------------+------------------+
| Intel         | DUAL RATE 1G/10G SFP+ LR (bailed)     | AFCT-701SDDZ-IN1 |
+---------------+---------------------------------------+------------------+

The following is a list of 3rd party SFP+ modules that have received some
testing. Not all modules are applicable to all devices.

+---------------+---------------------------------------+------------------+
| Supplier      | Type                                  | Part Numbers     |
+===============+=======================================+==================+
| Finisar       | SFP+ SR bailed, 10g single rate       | FTLX8571D3BCL    |
+---------------+---------------------------------------+------------------+
| Avago         | SFP+ SR bailed, 10g single rate       | AFBR-700SDZ      |
+---------------+---------------------------------------+------------------+
| Finisar       | SFP+ LR bailed, 10g single rate       | FTLX1471D3BCL    |
+---------------+---------------------------------------+------------------+
| Finisar       | DUAL RATE 1G/10G SFP+ SR (No Bail)    | FTLX8571D3QCV-IT |
+---------------+---------------------------------------+------------------+
| Avago         | DUAL RATE 1G/10G SFP+ SR (No Bail)    | AFBR-703SDZ-IN1  |
+---------------+---------------------------------------+------------------+
| Finisar       | DUAL RATE 1G/10G SFP+ LR (No Bail)    | FTLX1471D3QCV-IT |
+---------------+---------------------------------------+------------------+
| Avago         | DUAL RATE 1G/10G SFP+ LR (No Bail)    | AFCT-701SDZ-IN1  |
+---------------+---------------------------------------+------------------+
| Finisar       | 1000BASE-T SFP                        | FCLF8522P2BTL    |
+---------------+---------------------------------------+------------------+
| Avago         | 1000BASE-T                            | ABCU-5710RZ      |
+---------------+---------------------------------------+------------------+
| HP            | 1000BASE-SX SFP                       | 453153-001       |
+---------------+---------------------------------------+------------------+

82599-based adapters support all passive and active limiting direct attach
cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications.

SFP+ laser 제어와 82599 QSFP+

`ifconfig ethX down`을 실행할 때 SFP+ laser 끄기

`ifconfig ethX down`은 82599 기반 SFP+ fiber adapter의 laser를 끄고 `ifconfig ethX up`은 다시 켭니다.

대신 `ip link set down dev ethX`와 `ip link set up dev ethX`를 사용해 laser를 끄고 켤 수도 있습니다.

82599 기반 QSFP+ adapter

• Intel optics와 함께 제공된 82599 기반 Intel(R) Network Adapter는 Intel optics만 지원합니다.
• 4x10 Gbps 연결만 지원하고 1x40 Gbps 연결은 지원하지 않습니다. QSFP+ link partner도 4x10 Gbps로 구성해야 합니다.
• link speed 자동 감지를 지원하지 않습니다. link partner의 speed capability와 맞도록 10 Gbps 또는 1 Gbps를 직접 설정해야 하며, 설정이 틀리면 link가 형성되지 않습니다.
• Intel(R) Ethernet Converged Network Adapter X520-Q1은 아래 optics와 direct attach cable만 지원합니다.

82599 기반 QSFP+ adapter는 SFF-8436 v4.1 규격을 준수하는 모든 passive QSFP+ direct attach cable과 active limiting QSFP+ direct attach cable을 지원합니다.

Laser turns off for SFP+ when ifconfig ethX down
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"ifconfig ethX down" turns off the laser for 82599-based SFP+ fiber adapters.
"ifconfig ethX up" turns on the laser.
Alternatively, you can use "ip link set [down/up] dev ethX" to turn the
laser off and on.


82599-based QSFP+ Adapters
~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTES:
- If your 82599-based Intel(R) Network Adapter came with Intel optics, it only
supports Intel optics.
- 82599-based QSFP+ adapters only support 4x10 Gbps connections.  1x40 Gbps
connections are not supported. QSFP+ link partners must be configured for
4x10 Gbps.
- 82599-based QSFP+ adapters do not support automatic link speed detection.
The link speed must be configured to either 10 Gbps or 1 Gbps to match the link
partners speed capabilities. Incorrect speed configurations will result in
failure to link.
- Intel(R) Ethernet Converged Network Adapter X520-Q1 only supports the optics
and direct attach cables listed below.

+---------------+---------------------------------------+------------------+
| Supplier      | Type                                  | Part Numbers     |
+===============+=======================================+==================+
| Intel         | DUAL RATE 1G/10G QSFP+ SRL (bailed)   | E10GQSFPSR       |
+---------------+---------------------------------------+------------------+

82599-based QSFP+ adapters support all passive and active limiting QSFP+
direct attach cables that comply with SFF-8436 v4.1 specifications.

82598 기반 optics와 third-party 고지

82598 기반 adapter

• 교체식 optical module을 지원하는 Intel(R) Ethernet Network Adapter는 출고된 원래 module type만 지원합니다. 예를 들어 Intel(R) 10 Gigabit SR Dual Port Express Module은 SR optical module만 지원하며 다른 type을 꽂으면 driver가 load되지 않습니다.
• optical module의 hot swapping 또는 hot plugging은 지원하지 않습니다.
• single-speed 10 Gigabit module만 지원합니다.
• LAN on Motherboard(LOM)는 DA, SR 또는 LR module을 지원할 수 있습니다. 다른 type은 지원하지 않으므로 system 문서를 확인하십시오.

아래 SFP+ module과 direct attach cable은 일정 수준의 시험을 거쳤지만 모든 module이 모든 device에 적용되지는 않습니다.

82598 기반 adapter는 SFF-8431 v4.1과 SFF-8472 v10.4 규격을 준수하는 모든 passive direct attach cable을 지원합니다. active direct attach cable은 지원하지 않습니다.

위에 언급한 third-party optical module과 cable은 third-party 규격과 잠재적 호환성을 알리기 위해서만 나열한 것이며 Intel의 추천, 보증 또는 후원을 뜻하지 않습니다. Intel은 third-party 제품을 지지하거나 홍보하지 않습니다.

같거나 비슷한 설명의 optical module과 cable을 만드는 다른 manufacturer 또는 supplier가 있을 수 있습니다. 고객은 스스로 판단하고 충분히 조사해 선택한 third party에서 구매해야 하며, 제품과 device의 적합성 평가 및 vendor 선택에 전적으로 책임을 집니다.

위 optical module과 cable은 Intel이 보증하거나 지원하지 않습니다. Intel은 고객의 third-party 제품 판매·사용 또는 vendor 선택과 관련해 어떠한 책임도 지지 않으며 명시적·묵시적 보증을 모두 부인합니다.

82598-BASED ADAPTERS
~~~~~~~~~~~~~~~~~~~~
NOTES:
- Intel(r) Ethernet Network Adapters that support removable optical modules
only support their original module type (for example, the Intel(R) 10 Gigabit
SR Dual Port Express Module only supports SR optical modules). If you plug in
a different type of module, the driver will not load.
- Hot Swapping/hot plugging optical modules is not supported.
- Only single speed, 10 gigabit modules are supported.
- LAN on Motherboard (LOMs) may support DA, SR, or LR modules. Other module
types are not supported. Please see your system documentation for details.

The following is a list of SFP+ modules and direct attach cables that have
received some testing. Not all modules are applicable to all devices.

+---------------+---------------------------------------+------------------+
| Supplier      | Type                                  | Part Numbers     |
+===============+=======================================+==================+
| Finisar       | SFP+ SR bailed, 10g single rate       | FTLX8571D3BCL    |
+---------------+---------------------------------------+------------------+
| Avago         | SFP+ SR bailed, 10g single rate       | AFBR-700SDZ      |
+---------------+---------------------------------------+------------------+
| Finisar       | SFP+ LR bailed, 10g single rate       | FTLX1471D3BCL    |
+---------------+---------------------------------------+------------------+

82598-based adapters support all passive direct attach cables that comply with
SFF-8431 v4.1 and SFF-8472 v10.4 specifications. Active direct attach cables
are not supported.

Third party optic modules and cables referred to above are listed only for the
purpose of highlighting third party specifications and potential
compatibility, and are not recommendations or endorsements or sponsorship of
any third party's product by Intel. Intel is not endorsing or promoting
products made by any third party and the third party reference is provided
only to share information regarding certain optic modules and cables with the
above specifications. There may be other manufacturers or suppliers, producing
or supplying optic modules and cables with similar or matching descriptions.
Customers must use their own discretion and diligence to purchase optic
modules and cables from any third party of their choice. Customers are solely
responsible for assessing the suitability of the product and/or devices and
for the selection of the vendor for purchasing any product. THE OPTIC MODULES
AND CABLES REFERRED TO ABOVE ARE NOT WARRANTED OR SUPPORTED BY INTEL. INTEL
ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED
WARRANTY, RELATING TO SALE AND/OR USE OF SUCH THIRD PARTY PRODUCTS OR
SELECTION OF VENDOR BY CUSTOMERS.

max_vfs, VF 생성과 VLAN filter

Command line parameter

max_vfs

유효 범위: 1-63

이 parameter는 SR-IOV support를 추가하고 driver가 `max_vfs` 수까지 Virtual Function(VF)을 생성하게 합니다. 값이 0보다 크면 VMDq parameter도 1 이상으로 강제합니다.

`max_vfs`는 kernel 3.7.x 이하에서만 사용합니다. kernel 3.8.x 이상에서는 sysfs로 VF를 활성화합니다. Red Hat distribution에서도 6.6 이하는 이 parameter를 사용하지만 6.7 이상은 sysfs를 사용합니다.

#echo $num_vf_enabled > /sys/class/net/$dev/device/sriov_numvfs // enable VFs
#echo 0 > /sys/class/net/$dev/device/sriov_numvfs               //disable VFs

driver parameter는 position 기준으로 대응합니다. dual-port adapter 또는 여러 adapter가 있는 system에서 port마다 N개의 VF가 필요하면 port별 값을 comma로 구분해 지정해야 합니다.

modprobe ixgbe max_vfs=4
modprobe ixgbe max_vfs=2,4

첫 command는 첫 port에 VF 4개를 생성합니다. 둘째 command는 첫 port에 VF 2개, 둘째 port에 VF 4개를 생성합니다.

system configuration과 slot 수 등에 따라 command line의 position을 모든 경우에 예측할 수 없으므로 이 parameter로 driver를 load할 때 주의해야 합니다.

device와 driver 어느 쪽도 VF가 config space에 mapping되는 방식을 제어하지 않습니다. bus layout은 operating system마다 다르며 지원되는 OS에서는 sysfs로 mapping을 확인할 수 있습니다.

SR-IOV 또는 VMDq mode가 켜져 있으면 hardware VLAN filtering과 VLAN tag stripping/insertion이 계속 활성화됩니다. 새 VLAN filter를 추가하기 전에 기존 filter를 제거하십시오.

ip link set eth0 vf 0 vlan 100 // set VLAN 100 for VF 0
ip link set eth0 vf 0 vlan 0   // Delete VLAN 100
ip link set eth0 vf 0 vlan 200 // set a new VLAN 200 for VF 0

Command Line Parameters
=======================

max_vfs
-------
:Valid Range: 1-63

This parameter adds support for SR-IOV. It causes the driver to spawn up to
max_vfs worth of virtual functions.
If the value is greater than 0 it will also force the VMDq parameter to be 1 or
more.

NOTE: This parameter is only used on kernel 3.7.x and below. On kernel 3.8.x
and above, use sysfs to enable VFs. Also, for Red Hat distributions, this
parameter is only used on version 6.6 and older. For version 6.7 and newer, use
sysfs. For example::

  #echo $num_vf_enabled > /sys/class/net/$dev/device/sriov_numvfs // enable VFs
  #echo 0 > /sys/class/net/$dev/device/sriov_numvfs               //disable VFs

The parameters for the driver are referenced by position. Thus, if you have a
dual port adapter, or more than one adapter in your system, and want N virtual
functions per port, you must specify a number for each port with each parameter
separated by a comma. For example::

  modprobe ixgbe max_vfs=4

This will spawn 4 VFs on the first port.

::

  modprobe ixgbe max_vfs=2,4

This will spawn 2 VFs on the first port and 4 VFs on the second port.

NOTE: Caution must be used in loading the driver with these parameters.
Depending on your system configuration, number of slots, etc., it is impossible
to predict in all cases where the positions would be on the command line.

NOTE: Neither the device nor the driver control how VFs are mapped into config
space. Bus layout will vary by operating system. On operating systems that
support it, you can check sysfs to find the mapping.

NOTE: When either SR-IOV mode or VMDq mode is enabled, hardware VLAN filtering
and VLAN tag stripping/insertion will remain enabled. Please remove the old
VLAN filter before the new VLAN filter is added. For example,

::

  ip link set eth0 vf 0 vlan 100 // set VLAN 100 for VF 0
  ip link set eth0 vf 0 vlan 0   // Delete VLAN 100
  ip link set eth0 vf 0 vlan 200 // set a new VLAN 200 for VF 0

VF와 DCB resource 제약, SFP+·debug parameter

kernel 3.6부터 아래 제약 아래에서 `max_vfs`와 DCB 기능을 동시에 사용할 수 있습니다. kernel 3.6 이전에는 `max_vfs > 0`과 Priority Flow Control 및 Extended Transmission Selection을 사용하는 여러 traffic class의 동시 동작을 지원하지 않았습니다.

DCB가 켜지면 traffic은 NIC packet buffer인 여러 traffic class(TC)를 통해 송수신됩니다. VLAN tag의 0-7 priority가 어느 class에 속할지 결정합니다.

SR-IOV가 꺼져 있으면 각 TC는 receive/transmit descriptor queue pair 집합과 연결되며 pair 수는 hardware configuration에 따라 달라집니다. SR-IOV가 켜지면 descriptor queue pair를 pool로 묶고 PF와 각 VF에 receive/transmit descriptor queue pair pool 하나씩을 할당합니다.

여러 TC를 구성하면 각 pool에 TC마다 queue pair 하나가 들어갑니다. hardware에 TC 하나만 구성하면 pool 하나에 그 TC의 queue pair 여러 개가 들어갑니다.

VF가 구성되면 PF에도 pool 하나가 할당됩니다. 이때 PF는 DCB를 지원하지만 TC마다 queue pair 하나만 사용할 수 있습니다. VF가 0개이면 PF는 TC마다 queue pair 여러 개를 사용할 수 있습니다.

allow_unsupported_sfp

유효 범위: 0, 1. default: 0(disabled).

이 parameter는 module type을 driver가 알고 있다는 조건으로 82599 기반 adapter에서 지원·시험되지 않은 SFP+ module을 허용합니다.

debug

유효 범위: 0-16(`0=none`, `16=all`). default: 0.

system log에 표시되는 debug message level을 조정합니다.

With kernel 3.6, the driver supports the simultaneous usage of max_vfs and DCB
features, subject to the constraints described below. Prior to kernel 3.6, the
driver did not support the simultaneous operation of max_vfs greater than 0 and
the DCB features (multiple traffic classes utilizing Priority Flow Control and
Extended Transmission Selection).

When DCB is enabled, network traffic is transmitted and received through
multiple traffic classes (packet buffers in the NIC). The traffic is associated
with a specific class based on priority, which has a value of 0 through 7 used
in the VLAN tag. When SR-IOV is not enabled, each traffic class is associated
with a set of receive/transmit descriptor queue pairs. The number of queue
pairs for a given traffic class depends on the hardware configuration. When
SR-IOV is enabled, the descriptor queue pairs are grouped into pools. The
Physical Function (PF) and each Virtual Function (VF) is allocated a pool of
receive/transmit descriptor queue pairs. When multiple traffic classes are
configured (for example, DCB is enabled), each pool contains a queue pair from
each traffic class. When a single traffic class is configured in the hardware,
the pools contain multiple queue pairs from the single traffic class.

The number of VFs that can be allocated depends on the number of traffic
classes that can be enabled. The configurable number of traffic classes for
each enabled VF is as follows:
0 - 15 VFs = Up to 8 traffic classes, depending on device support
16 - 31 VFs = Up to 4 traffic classes
32 - 63 VFs = 1 traffic class

When VFs are configured, the PF is allocated one pool as well. The PF supports
the DCB features with the constraint that each traffic class will only use a
single queue pair. When zero VFs are configured, the PF can support multiple
queue pairs per traffic class.

allow_unsupported_sfp
---------------------
:Valid Range: 0,1
:Default Value: 0 (disabled)

This parameter allows unsupported and untested SFP+ modules on 82599-based
adapters, as long as the type of module is known to the driver.

debug
-----
:Valid Range: 0-16 (0=none,...,16=all)
:Default Value: 0

This parameter adjusts the level of debug messages displayed in the system
logs.

Ethernet Flow Control

추가 기능과 configuration

Flow Control

Ethernet Flow Control(IEEE 802.3x)은 ethtool로 ixgbe의 pause frame 수신과 송신을 켜도록 구성할 수 있습니다.

Tx가 켜지면 receive packet buffer가 미리 정한 threshold를 넘을 때 pause frame을 생성합니다. Rx가 켜지면 pause frame을 받았을 때 그 frame이 지정한 지연 시간 동안 transmit unit이 멈춥니다.

flow control을 지원하는 link partner가 필요합니다. Flow Control은 default로 활성화되어 있습니다.

Rx 또는 Tx Flow Control을 켜거나 끕니다.

ethtool -A eth? rx <on|off> tx <on|off>

auto-negotiation이 꺼져 있을 때만 이 command가 Flow Control 자체를 직접 켜거나 끕니다. auto-negotiation이 켜져 있으면 link partner와 협상할 때 사용할 parameter를 바꿉니다.

auto-negotiation을 켜거나 끕니다.

ethtool -s eth? autoneg <on|off>

Flow Control auto-negotiation은 link auto-negotiation의 일부입니다. device에 따라 auto-negotiation 설정을 변경할 수 없을 수도 있습니다.

82598 backplane card가 1 Gigabit mode로 들어갈 때는 Flow Control default가 off로 바뀝니다. 이 device에서 1 Gigabit mode의 Flow Control을 쓰면 transmit hang이 발생할 수 있습니다.

Additional Features and Configurations
======================================

Flow Control
------------
Ethernet Flow Control (IEEE 802.3x) can be configured with ethtool to enable
receiving and transmitting pause frames for ixgbe. When transmit is enabled,
pause frames are generated when the receive packet buffer crosses a predefined
threshold. When receive is enabled, the transmit unit will halt for the time
delay specified when a pause frame is received.

NOTE: You must have a flow control capable link partner.

Flow Control is enabled by default.

Use ethtool to change the flow control settings. To enable or disable Rx or
Tx Flow Control::

  ethtool -A eth? rx <on|off> tx <on|off>

Note: This command only enables or disables Flow Control if auto-negotiation is
disabled. If auto-negotiation is enabled, this command changes the parameters
used for auto-negotiation with the link partner.

To enable or disable auto-negotiation::

  ethtool -s eth? autoneg <on|off>

Note: Flow Control auto-negotiation is part of link auto-negotiation. Depending
on your device, you may not be able to change the auto-negotiation setting.

NOTE: For 82598 backplane cards entering 1 gigabit mode, flow control default
behavior is changed to off. Flow control in 1 gigabit mode on these devices can
lead to transmit hangs.

Intel Ethernet Flow Director

Intel(R) Ethernet Flow Director

Intel Ethernet Flow Director는 다음 작업을 수행합니다.

• receive packet을 flow에 따라 서로 다른 queue로 보냅니다.
• platform 안에서 flow routing을 세밀하게 제어합니다.
• flow와 CPU core를 대응시켜 flow affinity를 제공합니다.
• SFP mode에서만 유연한 flow classification과 load balancing을 위한 여러 parameter를 지원합니다.

Intel Ethernet Flow Director의 mask는 subnet mask와 반대 방식으로 동작합니다. 아래 command에서 filter에 실제로 기록되는 `src-ip`는 예상할 수 있는 `172.0.0.0`이 아니라 `0.4.1.2`입니다. `dst-ip`도 `172.0.0.0`이 아니라 `0.21.1.1`입니다.

#ethtool -N eth11 flow-type ip4 src-ip 172.4.1.2 m 255.0.0.0 dst-ip \
172.21.1.1 m 255.128.0.0 action 31

Intel Ethernet Flow Director를 켜거나 끕니다.

# ethtool -K ethX ntuple <on|off>

ntuple filter를 끄면 사용자가 구성한 모든 filter가 driver cache와 hardware에서 지워집니다. 다시 켤 때 필요한 filter를 모두 재등록해야 합니다.

packet을 queue 2로 보내는 filter는 `-U` 또는 `-N` switch로 추가합니다.

# ethtool -N ethX flow-type tcp4 src-ip 192.168.10.1 dst-ip \
192.168.10.2 src-port 2000 dst-port 2001 action 2 [loc 1]

현재 등록된 filter 목록은 다음 command로 확인합니다.

# ethtool <-u|-n> ethX

Intel(R) Ethernet Flow Director
-------------------------------
The Intel Ethernet Flow Director performs the following tasks:

- Directs receive packets according to their flows to different queues.
- Enables tight control on routing a flow in the platform.
- Matches flows and CPU cores for flow affinity.
- Supports multiple parameters for flexible flow classification and load
  balancing (in SFP mode only).

NOTE: Intel Ethernet Flow Director masking works in the opposite manner from
subnet masking. In the following command::

  #ethtool -N eth11 flow-type ip4 src-ip 172.4.1.2 m 255.0.0.0 dst-ip \
  172.21.1.1 m 255.128.0.0 action 31

The src-ip value that is written to the filter will be 0.4.1.2, not 172.0.0.0
as might be expected. Similarly, the dst-ip value written to the filter will be
0.21.1.1, not 172.0.0.0.

To enable or disable the Intel Ethernet Flow Director::

  # ethtool -K ethX ntuple <on|off>

When disabling ntuple filters, all the user programmed filters are flushed from
the driver cache and hardware. All needed filters must be re-added when ntuple
is re-enabled.

To add a filter that directs packet to queue 2, use -U or -N switch::

  # ethtool -N ethX flow-type tcp4 src-ip 192.168.10.1 dst-ip \
  192.168.10.2 src-port 2000 dst-port 2001 action 2 [loc 1]

To see the list of filters currently present::

  # ethtool <-u|-n> ethX

Sideband Perfect Filter

Sideband Perfect Filter는 지정한 특성과 일치하는 traffic을 특정 queue로 보냅니다. ethtool의 ntuple interface로 활성화하고 다음 형식으로 filter를 추가합니다.

ethtool -U <device> flow-type <type> src-ip <ip> dst-ip <ip> src-port <port> \
dst-port <port> action <queue>

filter를 삭제합니다.

ethtool -U <device> delete <N>

`<N>`은 active filter 전체를 출력할 때 표시되는 filter id입니다. filter를 추가할 때 `loc <N>`으로 지정했을 수도 있습니다.

아래 예시는 `192.168.0.1:5300`에서 `192.168.0.5:80`으로 전송되는 TCP traffic을 일치시켜 queue 7로 보냅니다.

ethtool -U enp130s0 flow-type tcp4 src-ip 192.168.0.1 dst-ip 192.168.0.5 \
src-port 5300 dst-port 80 action 7

같은 flow-type에 program하는 모든 filter는 동일한 matching input set을 사용해야 합니다. 다음 두 command는 모두 `src-ip`와 `src-port`를 사용하므로 허용됩니다.

ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.5 src-port 55 action 10

다음 두 command는 첫째가 `src-ip`, 둘째가 `dst-ip`를 지정하므로 허용되지 않으며 둘째 command가 error로 실패합니다.

ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
ethtool -U enp130s0 flow-type ip4 dst-ip 192.168.0.5 src-port 55 action 10

field는 같고 값만 다른 filter 여러 개는 program할 수 있습니다. 그러나 device 하나에서 서로 다른 matching field를 쓰는 TCP4 filter 두 개는 program할 수 없습니다. ixgbe driver는 field 일부만 일치시키는 partial mask도 지원하지 않습니다.

특정 VF로 traffic을 보내려면 `user-def` parameter를 사용합니다. 64-bit 값의 하위 32 bit는 queue number이고 그다음 8 bit는 VF를 나타냅니다. 0은 PF이므로 VF identifier에는 1의 offset이 적용됩니다.

... user-def 0x800000002 ...

`0x800000002`는 8에서 1을 뺀 Virtual Function 7의 queue 2로 traffic을 보내라는 뜻입니다.

이 filter는 내부 routing rule을 깨지 않습니다. 원래 지정한 VF로 전송될 수 없었던 traffic을 억지로 그 VF에 route하지도 않습니다.

Sideband Perfect Filters
------------------------
Sideband Perfect Filters are used to direct traffic that matches specified
characteristics. They are enabled through ethtool's ntuple interface. To add a
new filter use the following command::

  ethtool -U <device> flow-type <type> src-ip <ip> dst-ip <ip> src-port <port> \
  dst-port <port> action <queue>

Where:
  <device> - the ethernet device to program
  <type> - can be ip4, tcp4, udp4, or sctp4
  <ip> - the IP address to match on
  <port> - the port number to match on
  <queue> - the queue to direct traffic towards (-1 discards the matched traffic)

Use the following command to delete a filter::

  ethtool -U <device> delete <N>

Where <N> is the filter id displayed when printing all the active filters, and
may also have been specified using "loc <N>" when adding the filter.

The following example matches TCP traffic sent from 192.168.0.1, port 5300,
directed to 192.168.0.5, port 80, and sends it to queue 7::

  ethtool -U enp130s0 flow-type tcp4 src-ip 192.168.0.1 dst-ip 192.168.0.5 \
  src-port 5300 dst-port 80 action 7

For each flow-type, the programmed filters must all have the same matching
input set. For example, issuing the following two commands is acceptable::

  ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
  ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.5 src-port 55 action 10

Issuing the next two commands, however, is not acceptable, since the first
specifies src-ip and the second specifies dst-ip::

  ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
  ethtool -U enp130s0 flow-type ip4 dst-ip 192.168.0.5 src-port 55 action 10

The second command will fail with an error. You may program multiple filters
with the same fields, using different values, but, on one device, you may not
program two TCP4 filters with different matching fields.

Matching on a sub-portion of a field is not supported by the ixgbe driver, thus
partial mask fields are not supported.

To create filters that direct traffic to a specific Virtual Function, use the
"user-def" parameter. Specify the user-def as a 64 bit value, where the lower 32
bits represents the queue number, while the next 8 bits represent which VF.
Note that 0 is the PF, so the VF identifier is offset by 1. For example::

  ... user-def 0x800000002 ...

specifies to direct traffic to Virtual Function 7 (8 minus 1) into queue 2 of
that VF.

Note that these filters will not break internal routing rules, and will not
route traffic that otherwise would not have been sent to the specified Virtual
Function.

Jumbo frame, NBASE-T와 GRO

Jumbo Frame

MTU(Maximum Transmission Unit)를 default 1500보다 큰 값으로 바꾸면 jumbo frame support가 활성화됩니다.

ifconfig eth<x> mtu 9000 up
ip link set mtu 9000 dev eth<x>
ip link set up dev eth<x>

이 설정은 reboot 후 유지되지 않습니다. RHEL에서는 `/etc/sysconfig/network-scripts/ifcfg-eth<x>`, SLES에서는 `/etc/sysconfig/network/<config_file>`에 `MTU=9000`을 추가해 영구 적용합니다.

최대 jumbo MTU는 9710이며 최대 jumbo frame 크기 9728 bytes에 해당합니다.

driver는 jumbo packet 하나를 받기 위해 page-size buffer 여러 개를 사용하려고 합니다. receive packet을 allocate할 때 buffer starvation을 피하는 데 도움이 됩니다.

82599 기반 network connection에서 VF에 jumbo frame을 켜려면 PF에 먼저 jumbo frame을 켜야 합니다. VF MTU는 PF MTU보다 클 수 없습니다.

NBASE-T support

ixgbe driver는 일부 device에서 NBASE-T를 지원하지만, NBASE-T speed advertisement를 처리하지 못하는 고장 난 switch와의 호환성을 위해 default로 광고를 억제합니다. 지원 device에서 다음 command로 NBASE-T speed 광고를 켭니다.

ethtool -s eth? advertise 0x1800000001028

INTERFACES(5)를 쓰는 Linux system에서는 interface가 항상 NBASE-T support와 함께 올라오도록 `/etc/network/interfaces`의 pre-up command로 지정할 수 있습니다.

iface eth? inet dhcp
     pre-up ethtool -s eth? advertise 0x1800000001028 || true

Generic Receive Offload(GRO)

driver는 in-kernel software GRO 구현을 지원합니다. GRO는 Rx traffic을 더 큰 data chunk로 합쳐 큰 Rx load에서 CPU 사용량을 크게 낮출 수 있습니다.

GRO는 이전 LRO interface의 발전형이며 TCP 외 protocol도 합칠 수 있습니다. LRO에서 문제가 되는 bridging과 iSCSI configuration에도 안전하게 사용할 수 있습니다.

Jumbo Frames
------------
Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)
to a value larger than the default value of 1500.

Use the ifconfig command to increase the MTU size. For example, enter the
following where <x> is the interface number::

  ifconfig eth<x> mtu 9000 up

Alternatively, you can use the ip command as follows::

  ip link set mtu 9000 dev eth<x>
  ip link set up dev eth<x>

This setting is not saved across reboots. The setting change can be made
permanent by adding 'MTU=9000' to the file::

  /etc/sysconfig/network-scripts/ifcfg-eth<x> // for RHEL
  /etc/sysconfig/network/<config_file> // for SLES

NOTE: The maximum MTU setting for Jumbo Frames is 9710. This value coincides
with the maximum Jumbo Frames size of 9728 bytes.

NOTE: This driver will attempt to use multiple page sized buffers to receive
each jumbo packet. This should help to avoid buffer starvation issues when
allocating receive packets.

NOTE: For 82599-based network connections, if you are enabling jumbo frames in
a virtual function (VF), jumbo frames must first be enabled in the physical
function (PF). The VF MTU setting cannot be larger than the PF MTU.

NBASE-T Support
---------------
The ixgbe driver supports NBASE-T on some devices. However, the advertisement
of NBASE-T speeds is suppressed by default, to accommodate broken network
switches which cannot cope with advertised NBASE-T speeds. Use the ethtool
command to enable advertising NBASE-T speeds on devices which support it::

  ethtool -s eth? advertise 0x1800000001028

On Linux systems with INTERFACES(5), this can be specified as a pre-up command
in /etc/network/interfaces so that the interface is always brought up with
NBASE-T support, e.g.::

  iface eth? inet dhcp
       pre-up ethtool -s eth? advertise 0x1800000001028 || true

Generic Receive Offload, aka GRO
--------------------------------
The driver supports the in-kernel software implementation of GRO. GRO has
shown that by coalescing Rx traffic into larger chunks of data, CPU
utilization can be significantly reduced when under large Rx load. GRO is an
evolution of the previously-used LRO interface. GRO is able to coalesce
other protocols besides TCP. It's also safe to use with configurations that
are problematic for LRO, namely bridging and iSCSI.

DCB, ethtool과 FCoE

Data Center Bridging(DCB)

kernel은 TC0를 사용할 수 있다고 가정합니다. TC0를 사용할 수 없으면 device의 Priority Flow Control(PFC)을 끕니다. 이를 피하려면 switch에 DCB를 설정할 때 TC0가 활성화되어 있는지 확인하십시오.

DCB는 hardware에서 구현한 configurable Quality of Service입니다. VLAN priority tag(802.1p)를 이용해 traffic을 8개 priority로 분류합니다.

priority flow control(802.1Qbb)은 network stress 때 drop되는 packet 수를 제한하거나 없앨 수 있습니다. 각 priority에 bandwidth를 할당할 수 있으며 hardware level에서 802.1Qaz에 따라 강제됩니다.

adapter firmware는 각각 802.1AB와 802.1Qaz에 따른 LLDP와 DCBX protocol agent를 구현합니다. firmware DCBX agent는 willing mode로만 동작하며 DCBX를 지원하는 peer의 설정을 받아들일 수 있습니다.

`dcbtool`이나 `lldptool`을 통한 DCBX parameter의 software configuration은 지원하지 않습니다. ixgbe driver는 user space가 driver와 통신하고 port의 DCB configuration을 query할 수 있도록 DCB netlink interface layer를 구현합니다.

ethtool

driver configuration, 진단과 통계 표시에는 ethtool interface를 사용합니다. 이 기능에는 최신 ethtool이 필요하며 `https://www.kernel.org/pub/software/network/ethtool/`에서 받을 수 있습니다.

FCoE

ixgbe driver는 Fiber Channel over Ethernet(FCoE)과 DCB를 지원합니다. 이 code는 일반 driver 동작에 default로 영향을 주지 않습니다.

DCB와 FCoE configuration은 이 README의 범위를 벗어납니다. FCoE project 정보는 `http://www.open-fcoe.org/`, DCB 정보는 `[email protected]`를 참고하십시오.

Data Center Bridging (DCB)
--------------------------
NOTE:
The kernel assumes that TC0 is available, and will disable Priority Flow
Control (PFC) on the device if TC0 is not available. To fix this, ensure TC0 is
enabled when setting up DCB on your switch.

DCB is a configuration Quality of Service implementation in hardware. It uses
the VLAN priority tag (802.1p) to filter traffic. That means that there are 8
different priorities that traffic can be filtered into. It also enables
priority flow control (802.1Qbb) which can limit or eliminate the number of
dropped packets during network stress. Bandwidth can be allocated to each of
these priorities, which is enforced at the hardware level (802.1Qaz).

Adapter firmware implements LLDP and DCBX protocol agents as per 802.1AB and
802.1Qaz respectively. The firmware based DCBX agent runs in willing mode only
and can accept settings from a DCBX capable peer. Software configuration of
DCBX parameters via dcbtool/lldptool are not supported.

The ixgbe driver implements the DCB netlink interface layer to allow user-space
to communicate with the driver and query DCB configuration for the port.

ethtool
-------
The driver utilizes the ethtool interface for driver configuration and
diagnostics, as well as displaying statistical information. The latest ethtool
version is required for this functionality. Download it at:
https://www.kernel.org/pub/software/network/ethtool/

FCoE
----
The ixgbe driver supports Fiber Channel over Ethernet (FCoE) and Data Center
Bridging (DCB). This code has no default effect on the regular driver
operation. Configuring DCB and FCoE is outside the scope of this README. Refer
to http://www.open-fcoe.org/ for FCoE project information and contact
[email protected] for DCB information.

MAC·VLAN anti-spoofing과 IPsec offload

MAC 및 VLAN anti-spoofing 기능

악의적인 driver가 spoofed packet을 보내려 하면 hardware가 packet을 drop하므로 전송되지 않습니다.

spoof attempt를 알리는 interrupt가 PF driver에 전달됩니다. spoofed packet을 감지하면 PF driver는 `dmesg`로 볼 수 있는 다음 message를 system log에 기록합니다.

ixgbe ethX: ixgbe_spoof_check: n spoofed packets detected

여기서 `X`는 PF interface number이고 `n`은 spoofed packet 수입니다. 특정 VF에서는 이 기능을 끌 수 있습니다.

ip link set <pf dev> vf <vf id> spoofchk {off|on}

IPsec Offload

ixgbe driver는 IPsec Hardware Offload를 지원합니다. `ip xfrm ...`으로 Security Association(SA)을 만들 때 `offload` tag option을 사용하면 IPsec SA를 driver에 등록해 secure communication의 throughput을 높일 수 있습니다.

ixgbe VF에서도 offload를 지원하지만 VF를 `trusted`로 설정하고 private flag를 켜야 합니다.

ethtool --set-priv-flags eth<x> vf-ipsec on
ip link set eth<x> vf <y> trust on

MAC and VLAN anti-spoofing feature
----------------------------------
When a malicious driver attempts to send a spoofed packet, it is dropped by the
hardware and not transmitted.

An interrupt is sent to the PF driver notifying it of the spoof attempt. When a
spoofed packet is detected, the PF driver will send the following message to
the system log (displayed by the "dmesg" command)::

  ixgbe ethX: ixgbe_spoof_check: n spoofed packets detected

where "x" is the PF interface number; and "n" is number of spoofed packets.
NOTE: This feature can be disabled for a specific Virtual Function (VF)::

  ip link set <pf dev> vf <vf id> spoofchk {off|on}

IPsec Offload
-------------
The ixgbe driver supports IPsec Hardware Offload.  When creating Security
Associations with "ip xfrm ..." the 'offload' tag option can be used to
register the IPsec SA with the driver in order to get higher throughput in
the secure communications.

The offload is also supported for ixgbe's VFs, but the VF must be set as
'trusted' and the support must be enabled with::

  ethtool --set-priv-flags eth<x> vf-ipsec on
  ip link set eth<x> vf <y> trust on

알려진 문제와 지원

알려진 문제와 troubleshooting

64-bit Microsoft Windows Server 2012/R2 guest OS에서 SR-IOV 활성화

Linux KVM Hypervisor/VMM은 PCIe device를 VM에 직접 할당하는 기능을 지원합니다. 여기에는 기존 PCIe device와 Intel Ethernet Controller XL710 기반 SR-IOV-capable device가 포함됩니다.

지원

일반 정보는 Intel support website `https://www.intel.com/support/`를 참고하십시오.

지원되는 adapter와 kernel에서 공개된 source code 문제가 확인되면 문제와 관련한 구체적인 정보를 `[email protected]`로 보내십시오.

Known Issues/Troubleshooting
============================

Enabling SR-IOV in a 64-bit Microsoft Windows Server 2012/R2 guest OS
---------------------------------------------------------------------
Linux KVM Hypervisor/VMM supports direct assignment of a PCIe device to a VM.
This includes traditional PCIe devices, as well as SR-IOV-capable devices based
on the Intel Ethernet Controller XL710.


Support
=======
For general information, go to the Intel support website at:
https://www.intel.com/support/

If an issue is identified with the released source code on a supported kernel
with a supported adapter, email the specific information related to the issue
to [email protected].