← Documents Documentation/networking/devlink/mlx5.rst GitHub 원문 ↗

Linux 6.18.37 · Networking

mlx5 devlink support

mlx5의 자원·E-Switch·PCIe 매개변수와 TX/RX/펌웨어/VNIC 상태 보고자를 설명합니다.

Source pathDocumentation/networking/devlink/mlx5.rst
Source versionLinux v6.18.37
TranslationDUJINLABS 전문 번역 + 해설

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ====================
4 mlx5 devlink support
5 ====================
6
7 This document describes the devlink features implemented by the ``mlx5``
8 device driver.
9
10 Parameters
11 ==========
12
13 .. list-table:: Generic parameters implemented
14
15 * - Name
16 - Mode
17 - Validation
18 - Notes
19 * - ``enable_roce``
20 - driverinit
21 - Boolean
22 - If the device supports RoCE disablement, RoCE enablement state controls
23 device support for RoCE capability. Otherwise, the control occurs in the
24 driver stack. When RoCE is disabled at the driver level, only raw
25 ethernet QPs are supported.
26 * - ``io_eq_size``
27 - driverinit
28 - The range is between 64 and 4096.
29 -
30 * - ``event_eq_size``
31 - driverinit
32 - The range is between 64 and 4096.
33 -
34 * - ``max_macs``
35 - driverinit
36 - The range is between 1 and 2^31. Only power of 2 values are supported.
37 -
38 * - ``enable_sriov``
39 - permanent
40 - Boolean
41 - Applies to each physical function (PF) independently, if the device
42 supports it. Otherwise, it applies symmetrically to all PFs.
43 * - ``total_vfs``
44 - permanent
45 - The range is between 1 and a device-specific max.
46 - Applies to each physical function (PF) independently, if the device
47 supports it. Otherwise, it applies symmetrically to all PFs.
48
49 Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect
50
51 .. code-block:: bash
52
53 # setup parameters
54 devlink dev param set pci/0000:01:00.0 name enable_sriov value true cmode permanent
55 devlink dev param set pci/0000:01:00.0 name total_vfs value 8 cmode permanent
56
57 # Fw reset
58 devlink dev reload pci/0000:01:00.0 action fw_activate
59
60 # for PCI related config such as sriov PCI reset/rescan is required:
61 echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
62 echo 1 >/sys/bus/pci/rescan
63 grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
64
65 * - ``num_doorbells``
66 - driverinit
67 - This controls the number of channel doorbells used by the netdev. In all
68 cases, an additional doorbell is allocated and used for non-channel
69 communication (e.g. for PTP, HWS, etc.). Supported values are:
70
71 - 0: No channel-specific doorbells, use the global one for everything.
72 - [1, max_num_channels]: Spread netdev channels equally across these
73 doorbells.
74
75 The ``mlx5`` driver also implements the following driver-specific
76 parameters.
77
78 .. list-table:: Driver-specific parameters implemented
79 :widths: 5 5 5 85
80
81 * - Name
82 - Type
83 - Mode
84 - Description
85 * - ``flow_steering_mode``
86 - string
87 - runtime
88 - Controls the flow steering mode of the driver
89
90 * ``dmfs`` Device managed flow steering. In DMFS mode, the HW
91 steering entities are created and managed through firmware.
92 * ``smfs`` Software managed flow steering. In SMFS mode, the HW
93 steering entities are created and manage through the driver without
94 firmware intervention.
95 * ``hmfs`` Hardware managed flow steering. In HMFS mode, the driver
96 is configuring steering rules directly to the HW using Work Queues with
97 a special new type of WQE (Work Queue Element).
98
99 SMFS mode is faster and provides better rule insertion rate compared to
100 default DMFS mode.
101 * - ``fdb_large_groups``
102 - u32
103 - driverinit
104 - Control the number of large groups (size > 1) in the FDB table.
105
106 * The default value is 15, and the range is between 1 and 1024.
107 * - ``esw_multiport``
108 - Boolean
109 - runtime
110 - Control MultiPort E-Switch shared fdb mode.
111
112 An experimental mode where a single E-Switch is used and all the vports
113 and physical ports on the NIC are connected to it.
114
115 An example is to send traffic from a VF that is created on PF0 to an
116 uplink that is natively associated with the uplink of PF1
117
118 Note: Future devices, ConnectX-8 and onward, will eventually have this
119 as the default to allow forwarding between all NIC ports in a single
120 E-switch environment and the dual E-switch mode will likely get
121 deprecated.
122
123 Default: disabled
124 * - ``esw_port_metadata``
125 - Boolean
126 - runtime
127 - When applicable, disabling eswitch metadata can increase packet rate up
128 to 20% depending on the use case and packet sizes.
129
130 Eswitch port metadata state controls whether to internally tag packets
131 with metadata. Metadata tagging must be enabled for multi-port RoCE,
132 failover between representors and stacked devices. By default metadata is
133 enabled on the supported devices in E-switch. Metadata is applicable only
134 for E-switch in switchdev mode and users may disable it when NONE of the
135 below use cases will be in use:
136 1. HCA is in Dual/multi-port RoCE mode.
137 2. VF/SF representor bonding (Usually used for Live migration)
138 3. Stacked devices
139
140 When metadata is disabled, the above use cases will fail to initialize if
141 users try to enable them.
142
143 Note: Setting this parameter does not take effect immediately. Setting
144 must happen in legacy mode and eswitch port metadata takes effect after
145 enabling switchdev mode.
146 * - ``hairpin_num_queues``
147 - u32
148 - driverinit
149 - We refer to a TC NIC rule that involves forwarding as "hairpin".
150 Hairpin queues are mlx5 hardware specific implementation for hardware
151 forwarding of such packets.
152
153 Control the number of hairpin queues.
154 * - ``hairpin_queue_size``
155 - u32
156 - driverinit
157 - Control the size (in packets) of the hairpin queues.
158 * - ``pcie_cong_inbound_high``
159 - u16
160 - driverinit
161 - High threshold configuration for PCIe congestion events. The firmware
162 will send an event once device side inbound PCIe traffic went
163 above the configured high threshold for a long enough period (at least
164 200ms).
165
166 See pci_bw_inbound_high ethtool stat.
167
168 Units are 0.01 %. Accepted values are in range [0, 10000].
169 pcie_cong_inbound_low < pcie_cong_inbound_high.
170 Default value: 9000 (Corresponds to 90%).
171 * - ``pcie_cong_inbound_low``
172 - u16
173 - driverinit
174 - Low threshold configuration for PCIe congestion events. The firmware
175 will send an event once device side inbound PCIe traffic went
176 below the configured low threshold, only after having been previously in
177 a congested state.
178
179 See pci_bw_inbound_low ethtool stat.
180
181 Units are 0.01 %. Accepted values are in range [0, 10000].
182 pcie_cong_inbound_low < pcie_cong_inbound_high.
183 Default value: 7500.
184 * - ``pcie_cong_outbound_high``
185 - u16
186 - driverinit
187 - High threshold configuration for PCIe congestion events. The firmware
188 will send an event once device side outbound PCIe traffic went
189 above the configured high threshold for a long enough period (at least
190 200ms).
191
192 See pci_bw_outbound_high ethtool stat.
193
194 Units are 0.01 %. Accepted values are in range [0, 10000].
195 pcie_cong_outbound_low < pcie_cong_outbound_high.
196 Default value: 9000 (Corresponds to 90%).
197 * - ``pcie_cong_outbound_low``
198 - u16
199 - driverinit
200 - Low threshold configuration for PCIe congestion events. The firmware
201 will send an event once device side outbound PCIe traffic went
202 below the configured low threshold, only after having been previously in
203 a congested state.
204
205 See pci_bw_outbound_low ethtool stat.
206
207 Units are 0.01 %. Accepted values are in range [0, 10000].
208 pcie_cong_outbound_low < pcie_cong_outbound_high.
209 Default value: 7500.
210
211 * - ``cqe_compress_type``
212 - string
213 - permanent
214 - Configure which mechanism/algorithm should be used by the NIC that will
215 affect the rate (aggressiveness) of compressed CQEs depending on PCIe bus
216 conditions and other internal NIC factors. This mode affects all queues
217 that enable compression.
218 * ``balanced`` : Merges fewer CQEs, resulting in a moderate compression ratio but maintaining a balance between bandwidth savings and performance
219 * ``aggressive`` : Merges more CQEs into a single entry, achieving a higher compression rate and maximizing performance, particularly under high traffic loads
220
221 The ``mlx5`` driver supports reloading via ``DEVLINK_CMD_RELOAD``
222
223 Info versions
224 =============
225
226 The ``mlx5`` driver reports the following versions
227
228 .. list-table:: devlink info versions implemented
229 :widths: 5 5 90
230
231 * - Name
232 - Type
233 - Description
234 * - ``fw.psid``
235 - fixed
236 - Used to represent the board id of the device.
237 * - ``fw.version``
238 - stored, running
239 - Three digit major.minor.subminor firmware version number.
240
241 Health reporters
242 ================
243
244 tx reporter
245 -----------
246 The tx reporter is responsible for reporting and recovering of the following three error scenarios:
247
248 - tx timeout
249 Report on kernel tx timeout detection.
250 Recover by searching lost interrupts.
251 - tx error completion
252 Report on error tx completion.
253 Recover by flushing the tx queue and reset it.
254 - tx PTP port timestamping CQ unhealthy
255 Report too many CQEs never delivered on port ts CQ.
256 Recover by flushing and re-creating all PTP channels.
257
258 tx reporter also support on demand diagnose callback, on which it provides
259 real time information of its send queues status.
260
261 User commands examples:
262
263 - Diagnose send queues status::
264
265 $ devlink health diagnose pci/0000:82:00.0 reporter tx
266
267 .. note::
268 This command has valid output only when interface is up, otherwise the command has empty output.
269
270 - Show number of tx errors indicated, number of recover flows ended successfully,
271 is autorecover enabled and graceful period from last recover::
272
273 $ devlink health show pci/0000:82:00.0 reporter tx
274
275 rx reporter
276 -----------
277 The rx reporter is responsible for reporting and recovering of the following two error scenarios:
278
279 - rx queues' initialization (population) timeout
280 Population of rx queues' descriptors on ring initialization is done
281 in napi context via triggering an irq. In case of a failure to get
282 the minimum amount of descriptors, a timeout would occur, and
283 descriptors could be recovered by polling the EQ (Event Queue).
284 - rx completions with errors (reported by HW on interrupt context)
285 Report on rx completion error.
286 Recover (if needed) by flushing the related queue and reset it.
287
288 rx reporter also supports on demand diagnose callback, on which it
289 provides real time information of its receive queues' status.
290
291 - Diagnose rx queues' status and corresponding completion queue::
292
293 $ devlink health diagnose pci/0000:82:00.0 reporter rx
294
295 .. note::
296 This command has valid output only when interface is up. Otherwise, the command has empty output.
297
298 - Show number of rx errors indicated, number of recover flows ended successfully,
299 is autorecover enabled, and graceful period from last recover::
300
301 $ devlink health show pci/0000:82:00.0 reporter rx
302
303 fw reporter
304 -----------
305 The fw reporter implements `diagnose` and `dump` callbacks.
306 It follows symptoms of fw error such as fw syndrome by triggering
307 fw core dump and storing it into the dump buffer.
308 The fw reporter diagnose command can be triggered any time by the user to check
309 current fw status.
310
311 User commands examples:
312
313 - Check fw heath status::
314
315 $ devlink health diagnose pci/0000:82:00.0 reporter fw
316
317 - Read FW core dump if already stored or trigger new one::
318
319 $ devlink health dump show pci/0000:82:00.0 reporter fw
320
321 .. note::
322 This command can run only on the PF which has fw tracer ownership,
323 running it on other PF or any VF will return "Operation not permitted".
324
325 fw fatal reporter
326 -----------------
327 The fw fatal reporter implements `dump` and `recover` callbacks.
328 It follows fatal errors indications by CR-space dump and recover flow.
329 The CR-space dump uses vsc interface which is valid even if the FW command
330 interface is not functional, which is the case in most FW fatal errors.
331 The recover function runs recover flow which reloads the driver and triggers fw
332 reset if needed.
333 On firmware error, the health buffer is dumped into the dmesg. The log
334 level is derived from the error's severity (given in health buffer).
335
336 User commands examples:
337
338 - Run fw recover flow manually::
339
340 $ devlink health recover pci/0000:82:00.0 reporter fw_fatal
341
342 - Read FW CR-space dump if already stored or trigger new one::
343
344 $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal
345
346 .. note::
347 This command can run only on PF.
348
349 vnic reporter
350 -------------
351 The vnic reporter implements only the `diagnose` callback.
352 It is responsible for querying the vnic diagnostic counters from fw and displaying
353 them in realtime.
354
355 Description of the vnic counters:
356
357 - total_error_queues
358 number of queues in an error state due to
359 an async error or errored command.
360 - send_queue_priority_update_flow
361 number of QP/SQ priority/SL update events.
362 - cq_overrun
363 number of times CQ entered an error state due to an overflow.
364 - async_eq_overrun
365 number of times an EQ mapped to async events was overrun.
366 - comp_eq_overrun
367 number of times an EQ mapped to completion events was
368 overrun.
369 - quota_exceeded_command
370 number of commands issued and failed due to quota exceeded.
371 - invalid_command
372 number of commands issued and failed dues to any reason other than quota
373 exceeded.
374 - nic_receive_steering_discard
375 number of packets that completed RX flow
376 steering but were discarded due to a mismatch in flow table.
377 - generated_pkt_steering_fail
378 number of packets generated by the VNIC experiencing unexpected steering
379 failure (at any point in steering flow).
380 - handled_pkt_steering_fail
381 number of packets handled by the VNIC experiencing unexpected steering
382 failure (at any point in steering flow owned by the VNIC, including the FDB
383 for the eswitch owner).
384 - icm_consumption
385 amount of Interconnect Host Memory (ICM) consumed by the vnic in
386 granularity of 4KB. ICM is host memory allocated by SW upon HCA request
387 and is used for storing data structures that control HCA operation.
388 - bar_uar_access
389 number of WRITE or READ access operations to the UAR on the PCIe BAR.
390 - odp_local_triggered_page_fault
391 number of locally-triggered page-faults due to ODP.
392 - odp_remote_triggered_page_fault
393 number of remotly-triggered page-faults due to ODP.
394
395 User commands examples:
396
397 - Diagnose PF/VF vnic counters::
398
399 $ devlink health diagnose pci/0000:82:00.1 reporter vnic
400
401 - Diagnose representor vnic counters (performed by supplying devlink port of the
402 representor, which can be obtained via devlink port command)::
403
404 $ devlink health diagnose pci/0000:82:00.1/65537 reporter vnic
405
406 .. note::
407 This command can run over all interfaces such as PF/VF and representor ports.
408

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.