Meta Platforms Host Network Interface

펌웨어 구성 요소와 버전

이 문서는 `GPL-2.0+` 라이선스를 따릅니다.

Meta Platforms Host Network Interface

펌웨어 버전

fbnic 플래시에는 하나의 PLDM 이미지가 제공하는 세 구성 요소가 저장됩니다.

플래시에는 세 구성 요소의 복사본을 두 세트 저장합니다. 새 펌웨어가 부팅하지 못하면 자동으로 이전 버전으로 되돌아갈 수 있습니다. 두 세트의 버전은 running과 stored로 제공하지만, Linux 드라이버가 제어권을 넘겨받은 뒤 실행하지 않는 `undi`는 stored 버전만 표시합니다.

`devlink dev info`는 세 구성 요소의 버전과 빌드의 hg commit hash를 별도 항목으로 보여 줍니다.

.. SPDX-License-Identifier: GPL-2.0+

=====================================
Meta Platforms Host Network Interface
=====================================

Firmware Versions
-----------------

fbnic has three components stored on the flash which are provided in one PLDM
image:

1. fw - The control firmware used to view and modify firmware settings, request
   firmware actions, and retrieve firmware counters outside of the data path.
   This is the firmware which fbnic_fw.c interacts with.
2. bootloader - The firmware which validate firmware security and control basic
   operations including loading and updating the firmware. This is also known
   as the cmrt firmware.
3. undi - This is the UEFI driver which is based on the Linux driver.

fbnic stores two copies of these three components on flash. This allows fbnic
to fall back to an older version of firmware automatically in case firmware
fails to boot. Version information for both is provided as running and stored.
The undi is only provided in stored as it is not actively running once the Linux
driver takes over.

devlink dev info provides version information for all three components. In
addition to the version the hg commit hash of the build is included as a
separate entry.

Configuration

Ring parameter와 큐 구조

구성: Ringparams (`ethtool -g` / `-G`)

fbnic의 completion ring 하나마다 host→device submission ring 두 개가 대응합니다. 상위 소프트웨어가 말하는 RX 또는 TX queue 하나는 이 세 ring 객체의 조합입니다.

• `ethtool rx`는 completion ring 크기에 대응합니다.
• `rx-mini`는 Header Page Queue(HPQ)에 대응합니다.
• `rx-jumbo`는 Payload Page Queue(PPQ)에 대응합니다.

TX에서는 submission ring 둘 모두 패킷을 게시할 수 있고 completion ring 하나가 두 ring의 완료를 전달합니다. fbnic은 하나를 일반 stack traffic에, 다른 하나를 XDP frame에 씁니다. `ethtool tx`는 submission ring 둘과 completion ring의 크기를 함께 제어합니다.

HPQ와 PPQ의 각 entry는 4 KiB 메모리 할당 하나에 해당합니다. 나머지 ring entry의 단위는 8바이트 descriptor입니다. 작은 패킷 여러 개가 페이지 하나에 들어갈 수 있으므로 submission/completion ring의 이상적인 비율은 workload에 따라 달라집니다.

-------------

Ringparams (ethtool -g / -G)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

fbnic has two submission (host -> device) rings for every completion
(device -> host) ring. The three ring objects together form a single
"queue" as used by higher layer software (a Rx, or a Tx queue).

For Rx the two submission rings are used to pass empty pages to the NIC.
Ring 0 is the Header Page Queue (HPQ), NIC will use its pages to place
L2-L4 headers (or full frames if frame is not header-data split).
Ring 1 is the Payload Page Queue (PPQ) and used for packet payloads.
The completion ring is used to receive packet notifications / metadata.
ethtool ``rx`` ringparam maps to the size of the completion ring,
``rx-mini`` to the HPQ, and ``rx-jumbo`` to the PPQ.

For Tx both submission rings can be used to submit packets, the completion
ring carries notifications for both. fbnic uses one of the submission
rings for normal traffic from the stack and the second one for XDP frames.
ethtool ``tx`` ringparam controls both the size of the submission rings
and the completion ring.

Every single entry on the HPQ and PPQ (``rx-mini``, ``rx-jumbo``)
corresponds to 4kB of allocated memory, while entries on the remaining
rings are in units of descriptors (8B). The ideal ratio of submission
and completion ring sizes will depend on the workload, as for small packets
multiple packets will fit into a single page.

Upgrading Firmware

펌웨어 업데이트와 health reporter

펌웨어 업그레이드

fbnic은 서명된 PLDM 이미지를 `devlink dev flash`로 플래시에 기록합니다. 플래시 작업은 장치 동작을 중단하지 않습니다.

호스트 부팅 시 최신 UEFI 드라이버가 자동으로 사용되므로 별도 활성화가 필요 없습니다. 새 제어 펌웨어를 실행하려면 firmware activation이 필요하고, `cmrt` 펌웨어는 NIC 전원을 껐다 켜야 활성화됩니다.

Health reporter

`fw` diagnose는 최근 heartbeat의 uptime을 보여 주며 uptime이 감소했는지 확인해 crash를 판정합니다. OTP memory는 ECC로 보호됩니다.

------------------

fbnic supports updating firmware using signed PLDM images with devlink dev
flash. PLDM images are written into the flash. Flashing does not interrupt
the operation of the device.

On host boot the latest UEFI driver is always used, no explicit activation
is required. Firmware activation is required to run new control firmware. cmrt
firmware can only be activated by power cycling the NIC.

Health reporters
----------------

fw reporter
~~~~~~~~~~~

The ``fw`` health reporter tracks FW crashes. Dumping the reporter will
show the core dump of the most recent FW crash, and if no FW crash has
happened since power cycle - a snapshot of the FW memory. Diagnose callback
shows FW uptime based on the most recently received heartbeat message
(the crashes are detected by checking if uptime goes down).

otp reporter
~~~~~~~~~~~~

OTP memory ("fuses") are used for secure boot and anti-rollback
protection. The OTP memory is ECC protected, ECC errors indicate
either manufacturing defect or part deteriorating with age.

Statistics

MAC, RX buffer, parser와 DMA 통계

통계

----------

TX MAC Interface
~~~~~~~~~~~~~~~~

 - ``ptp_illegal_req``: packets sent to the NIC with PTP request bit set but routed to BMC/FW
 - ``ptp_good_ts``: packets successfully routed to MAC with PTP request bit set
 - ``ptp_bad_ts``: packets destined for MAC with PTP request bit set but aborted because of some error (e.g., DMA read error)

TX Extension (TEI) Interface (TTI)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 - ``tti_cm_drop``: control messages dropped at the TX Extension (TEI) Interface because of credit starvation
 - ``tti_frame_drop``: packets dropped at the TX Extension (TEI) Interface because of credit starvation
 - ``tti_tbi_drop``: packets dropped at the TX BMC Interface (TBI) because of credit starvation

RXB (RX Buffer) Enqueue
~~~~~~~~~~~~~~~~~~~~~~~

 - ``rxb_integrity_err[i]``: frames enqueued with integrity errors (e.g., multi-bit ECC errors) on RXB input i
 - ``rxb_mac_err[i]``: frames enqueued with MAC end-of-frame errors (e.g., bad FCS) on RXB input i
 - ``rxb_parser_err[i]``: frames experienced RPC parser errors
 - ``rxb_frm_err[i]``: frames experienced signaling errors (e.g., missing end-of-packet/start-of-packet) on RXB input i
 - ``rxb_drbo[i]_frames``: frames received at RXB input i
 - ``rxb_drbo[i]_bytes``: bytes received at RXB input i

RXB (RX Buffer) FIFO
~~~~~~~~~~~~~~~~~~~~

 - ``rxb_fifo[i]_drop``: transitions into the drop state on RXB pool i
 - ``rxb_fifo[i]_dropped_frames``: frames dropped on RXB pool i
 - ``rxb_fifo[i]_ecn``: transitions into the ECN mark state on RXB pool i
 - ``rxb_fifo[i]_level``: current occupancy of RXB pool i

RXB (RX Buffer) Dequeue
~~~~~~~~~~~~~~~~~~~~~~~

   - ``rxb_intf[i]_frames``: frames sent to the output i
   - ``rxb_intf[i]_bytes``: bytes sent to the output i
   - ``rxb_pbuf[i]_frames``: frames sent to output i from the perspective of internal packet buffer
   - ``rxb_pbuf[i]_bytes``: bytes sent to output i from the perspective of internal packet buffer

RPC (Rx parser)
~~~~~~~~~~~~~~~

 - ``rpc_unkn_etype``: frames containing unknown EtherType
 - ``rpc_unkn_ext_hdr``: frames containing unknown IPv6 extension header
 - ``rpc_ipv4_frag``: frames containing IPv4 fragment
 - ``rpc_ipv6_frag``: frames containing IPv6 fragment
 - ``rpc_ipv4_esp``: frames with IPv4 ESP encapsulation
 - ``rpc_ipv6_esp``: frames with IPv6 ESP encapsulation
 - ``rpc_tcp_opt_err``: frames which encountered TCP option parsing error
 - ``rpc_out_of_hdr_err``: frames where header was larger than parsable region
 - ``ovr_size_err``: oversized frames

Hardware Queues
~~~~~~~~~~~~~~~

1. RX DMA Engine:

 - ``rde_[i]_pkt_err``: packets with MAC EOP, RPC parser, RXB truncation, or RDE frame truncation errors. These error are flagged in the packet metadata because of cut-through support but the actual drop happens once PCIE/RDE is reached.
 - ``rde_[i]_pkt_cq_drop``: packets dropped because RCQ is full
 - ``rde_[i]_pkt_bdq_drop``: packets dropped because HPQ or PPQ ran out of host buffer

PCIe

PCIe 통계와 XDP 길이 오류

PCIe

fbnic은 debugfs의 `pcie_stats`로 PCIe hardware performance 통계를 노출합니다. transaction 동작과 잠재적 성능 병목을 파악할 수 있습니다.

XDP 길이 오류

fragment를 지원하지 않는 XDP 프로그램에서는 MTU가 버퍼 하나에 들어가도록 fbnic이 확인합니다. 너무 큰 frame이 들어와 조각나면 버리고 다음 netlink 카운터를 갱신합니다.

• `rx-length`: 연결된 XDP 프로그램이 fragmentation을 지원하지 않아 버린 frame 수
• `rx-errors`: 인터페이스가 받은 오류 packet의 전체 수

~~~~

The fbnic driver exposes PCIe hardware performance statistics through debugfs
(``pcie_stats``). These statistics provide insights into PCIe transaction
behavior and potential performance bottlenecks.

1. PCIe Transaction Counters:

   These counters track PCIe transaction activity:
        - ``pcie_ob_rd_tlp``: Outbound read Transaction Layer Packets count
        - ``pcie_ob_rd_dword``: DWORDs transferred in outbound read transactions
        - ``pcie_ob_wr_tlp``: Outbound write Transaction Layer Packets count
        - ``pcie_ob_wr_dword``: DWORDs transferred in outbound write
          transactions
        - ``pcie_ob_cpl_tlp``: Outbound completion TLP count
        - ``pcie_ob_cpl_dword``: DWORDs transferred in outbound completion TLPs

2. PCIe Resource Monitoring:

   These counters indicate PCIe resource exhaustion events:
        - ``pcie_ob_rd_no_tag``: Read requests dropped due to tag unavailability
        - ``pcie_ob_rd_no_cpl_cred``: Read requests dropped due to completion
          credit exhaustion
        - ``pcie_ob_rd_no_np_cred``: Read requests dropped due to non-posted
          credit exhaustion

XDP Length Error:
~~~~~~~~~~~~~~~~~

For XDP programs without frags support, fbnic tries to make sure that MTU fits
into a single buffer. If an oversized frame is received and gets fragmented,
it is dropped and the following netlink counters are updated

   - ``rx-length``: number of frames dropped due to lack of fragmentation
     support in the attached XDP program
   - ``rx-errors``: total number of packets with errors received on the interface