요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
Port counter와 hardware counter
sysfs-class-infiniband:85-216Link error, packet·octet, discard·wait counter와 hardware counter update lifespan을 다룹니다.
GID, MAD와 verbs ABI
sysfs-class-infiniband:219-284RoCE GID의 netdev·type, umad·issm port mapping과 uverbs ABI version을 다룹니다.
Mellanox mthca·mlx4와 IOV
sysfs-class-infiniband:287-400HCA identity와 SR-IOV GID·P_Key paravirtualization, VF QP0 SMI 제어를 다룹니다.
Driver별 HCA 확장
sysfs-class-infiniband:403-784cxgb4, qib, mlx5, usnic, ocrdma, hfi1, qedr, PVRDMA, bnxt_re의 board·context·congestion·lane mapping·SDMA attribute를 다룹니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
1
sysfs interface common for all infiniband devices
2
-------------------------------------------------
4
What: /sys/class/infiniband/<device>/node_type
5
What: /sys/class/infiniband/<device>/node_guid
6
What: /sys/class/infiniband/<device>/sys_image_guid
7
Date: Apr, 2005
8
KernelVersion: v2.6.12
9
Contact: [email protected]
10
Description:
11
=============== ===========================================
12
node_type: (RO) Node type (CA, RNIC, usNIC, usNIC UDP,
13
switch or router)
15
node_guid: (RO) Node GUID
17
sys_image_guid: (RO) System image GUID
18
=============== ===========================================
21
What: /sys/class/infiniband/<device>/node_desc
22
Date: Feb, 2006
23
KernelVersion: v2.6.17
24
Contact: [email protected]
25
Description:
26
(RW) Update the node description with information such as the
27
node's hostname, so that IB network management software can tie
28
its view to the real world.
31
What: /sys/class/infiniband/<device>/fw_ver
32
Date: Jun, 2016
33
KernelVersion: v4.10
34
Contact: [email protected]
35
Description:
36
(RO) Display firmware version
39
What: /sys/class/infiniband/<device>/ports/<port-num>/lid
40
What: /sys/class/infiniband/<device>/ports/<port-num>/rate
41
What: /sys/class/infiniband/<device>/ports/<port-num>/lid_mask_count
42
What: /sys/class/infiniband/<device>/ports/<port-num>/sm_sl
43
What: /sys/class/infiniband/<device>/ports/<port-num>/sm_lid
44
What: /sys/class/infiniband/<device>/ports/<port-num>/state
45
What: /sys/class/infiniband/<device>/ports/<port-num>/phys_state
46
What: /sys/class/infiniband/<device>/ports/<port-num>/cap_mask
47
Date: Apr, 2005
48
KernelVersion: v2.6.12
49
Contact: [email protected]
50
Description:
52
=============== ===============================================
53
lid: (RO) Port LID
55
rate: (RO) Port data rate (active width * active
56
speed)
58
lid_mask_count: (RO) Port LID mask count
60
sm_sl: (RO) Subnet manager SL for port's subnet
62
sm_lid: (RO) Subnet manager LID for port's subnet
64
state: (RO) Port state (DOWN, INIT, ARMED, ACTIVE or
65
ACTIVE_DEFER)
67
phys_state: (RO) Port physical state (Sleep, Polling,
68
LinkUp, etc)
70
cap_mask: (RO) Port capability mask. 2 bits here are
71
settable- IsCommunicationManagementSupported
72
(set when CM module is loaded) and IsSM (set
73
via open of issmN file).
74
=============== ===============================================
77
What: /sys/class/infiniband/<device>/ports/<port-num>/link_layer
78
Date: Oct, 2010
79
KernelVersion: v2.6.37
80
Contact: [email protected]
81
Description:
82
(RO) Link layer type information (Infiniband or Ethernet type)
85
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/symbol_error
86
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_errors
87
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_remote_physical_errors
88
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_switch_relay_errors
89
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/link_error_recovery
90
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_constraint_errors
91
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_contraint_errors
92
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/local_link_integrity_errors
93
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/excessive_buffer_overrun_errors
94
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_data
95
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_data
96
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_packets
97
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_rcv_packets
98
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/unicast_rcv_packets
99
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/unicast_xmit_packets
100
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/multicast_rcv_packets
101
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/multicast_xmit_packets
102
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/link_downed
103
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_discards
104
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/VL15_dropped
105
What: /sys/class/infiniband/<device>/ports/<port-num>/counters/port_xmit_wait
106
Date: Apr, 2005
107
KernelVersion: v2.6.12
108
Contact: [email protected]
109
Description:
110
**Errors info**:
112
symbol_error: (RO) Total number of minor link errors detected on
113
one or more physical lanes.
115
port_rcv_errors : (RO) Total number of packets containing an
116
error that were received on the port.
118
port_rcv_remote_physical_errors : (RO) Total number of packets
119
marked with the EBP delimiter received on the port.
121
port_rcv_switch_relay_errors : (RO) Total number of packets
122
received on the port that were discarded because they could not
123
be forwarded by the switch relay.
125
link_error_recovery: (RO) Total number of times the Port
126
Training state machine has successfully completed the link error
127
recovery process.
129
port_xmit_constraint_errors: (RO) Total number of packets not
130
transmitted from the switch physical port due to outbound raw
131
filtering or failing outbound partition or IP version check.
133
port_rcv_constraint_errors: (RO) Total number of packets
134
received on the switch physical port that are discarded due to
135
inbound raw filtering or failing inbound partition or IP version
136
check.
138
local_link_integrity_errors: (RO) The number of times that the
139
count of local physical errors exceeded the threshold specified
140
by LocalPhyErrors
142
excessive_buffer_overrun_errors: (RO) This counter, indicates an
143
input buffer overrun. It indicates possible misconfiguration of
144
a port, either by the Subnet Manager (SM) or by user
145
intervention. It can also indicate hardware issues or extremely
146
poor link signal integrity
148
**Data info**:
150
port_xmit_data: (RO) Total number of data octets, divided by 4
151
(lanes), transmitted on all VLs. This is 64 bit counter
153
port_rcv_data: (RO) Total number of data octets, divided by 4
154
(lanes), received on all VLs. This is 64 bit counter.
156
port_xmit_packets: (RO) Total number of packets transmitted on
157
all VLs from this port. This may include packets with errors.
158
This is 64 bit counter.
160
port_rcv_packets: (RO) Total number of packets (this may include
161
packets containing Errors. This is 64 bit counter.
163
link_downed: (RO) Total number of times the Port Training state
164
machine has failed the link error recovery process and downed
165
the link.
167
unicast_rcv_packets: (RO) Total number of unicast packets,
168
including unicast packets containing errors.
170
unicast_xmit_packets: (RO) Total number of unicast packets
171
transmitted on all VLs from the port. This may include unicast
172
packets with errors.
174
multicast_rcv_packets: (RO) Total number of multicast packets,
175
including multicast packets containing errors.
177
multicast_xmit_packets: (RO) Total number of multicast packets
178
transmitted on all VLs from the port. This may include multicast
179
packets with errors.
181
**Misc info**:
183
port_xmit_discards: (RO) Total number of outbound packets
184
discarded by the port because the port is down or congested.
186
VL15_dropped: (RO) Number of incoming VL15 packets dropped due
187
to resource limitations (e.g., lack of buffers) of the port.
189
port_xmit_wait: (RO) The number of ticks during which the port
190
had data to transmit but no data was sent during the entire tick
191
(either because of insufficient credits or because of lack of
192
arbitration).
194
Each of these files contains the corresponding value from the
195
port's Performance Management PortCounters attribute, as
196
described in the InfiniBand Architecture Specification.
199
What: /sys/class/infiniband/<device-name>/hw_counters/lifespan
200
What: /sys/class/infiniband/<device-name>/ports/<port-num>/hw_counters/lifespan
201
Date: May, 2016
202
KernelVersion: 4.6
203
Contact: [email protected]
204
Description:
205
The optional "hw_counters" subdirectory can be under either the
206
parent device or the port subdirectories or both. If present,
207
there are a list of counters provided by the hardware. They may
208
match some of the counters in the counters directory, but they
209
often include many other counters. In addition to the various
210
counters, there will be a file named "lifespan" that configures
211
how frequently the core should update the counters when they are
212
being accessed (counters are not updated if they are not being
213
accessed). The lifespan is in milliseconds and defaults to 10
214
unless set to something else by the driver. Users may echo a
215
value between 0-10000 to the lifespan file to set the length
216
of time between updates in milliseconds.
219
What: /sys/class/infiniband/<hca>/ports/<port-number>/gid_attrs/ndevs/<gid-index>
220
Date: November 29, 2015
221
KernelVersion: 4.4.0
222
Contact: [email protected]
223
Description: The net-device's name associated with the GID resides
224
at index <gid-index>.
226
What: /sys/class/infiniband/<hca>/ports/<port-number>/gid_attrs/types/<gid-index>
227
Date: November 29, 2015
228
KernelVersion: 4.4.0
229
Contact: [email protected]
230
Description: The RoCE type of the associated GID resides at index <gid-index>.
231
This could either be "IB/RoCE v1" for IB and RoCE v1 based GIDs
232
or "RoCE v2" for RoCE v2 based GIDs.
235
What: /sys/class/infiniband_mad/umad<N>/ibdev
236
What: /sys/class/infiniband_mad/umad<N>/port
237
What: /sys/class/infiniband_mad/issm<N>/ibdev
238
What: /sys/class/infiniband_mad/issm<N>/port
239
Date: Apr, 2005
240
KernelVersion: v2.6.12
241
Contact: [email protected]
242
Description:
243
Each port of each InfiniBand device has a "umad" device and an
244
"issm" device attached. For example, a two-port HCA will have
245
two umad devices and two issm devices, while a switch will have
246
one device of each type (for switch port 0).
248
======= =====================================
249
ibdev: (RO) Show Infiniband (IB) device name
251
port: (RO) Display port number
252
======= =====================================
255
What: /sys/class/infiniband_mad/abi_version
256
Date: Apr, 2005
257
KernelVersion: v2.6.12
258
Contact: [email protected]
259
Description:
260
(RO) Value is incremented if any changes are made that break
261
userspace ABI compatibility of umad & issm devices.
264
What: /sys/class/infiniband_verbs/uverbs<N>/ibdev
265
What: /sys/class/infiniband_verbs/uverbs<N>/abi_version
266
Date: Sept, 2005
267
KernelVersion: v2.6.14
268
Contact: [email protected]
269
Description:
270
=============== ===========================================
271
ibdev: (RO) Display Infiniband (IB) device name
273
abi_version: (RO) Show ABI version of IB device specific
274
interfaces.
275
=============== ===========================================
278
What: /sys/class/infiniband_verbs/abi_version
279
Date: Sep, 2005
280
KernelVersion: v2.6.14
281
Contact: [email protected]
282
Description:
283
(RO) Value is incremented if any changes are made that break
284
userspace ABI compatibility of uverbs devices.
287
sysfs interface for Mellanox IB HCA low-level driver (mthca)
288
------------------------------------------------------------
290
What: /sys/class/infiniband/mthcaX/hw_rev
291
What: /sys/class/infiniband/mthcaX/hca_type
292
What: /sys/class/infiniband/mthcaX/board_id
293
Date: Apr, 2005
294
KernelVersion: v2.6.12
295
Contact: [email protected]
296
Description:
297
=============== ================================================
298
hw_rev: (RO) Hardware revision number
300
hca_type: (RO) Host Channel Adapter type: MT23108, MT25208
301
(MT23108 compat mode), MT25208 or MT25204
303
board_id: (RO) Manufacturing board ID
304
=============== ================================================
307
sysfs interface for Mellanox ConnectX HCA IB driver (mlx4)
308
----------------------------------------------------------
310
What: /sys/class/infiniband/mlx4_X/hw_rev
311
What: /sys/class/infiniband/mlx4_X/hca_type
312
What: /sys/class/infiniband/mlx4_X/board_id
313
Date: Sep, 2007
314
KernelVersion: v2.6.24
315
Contact: [email protected]
316
Description:
317
=============== ===============================
318
hw_rev: (RO) Hardware revision number
320
hca_type: (RO) Host channel adapter type
322
board_id: (RO) Manufacturing board ID
323
=============== ===============================
326
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/gids/<n>
327
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/admin_guids/<n>
328
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/pkeys/<n>
329
What: /sys/class/infiniband/mlx4_X/iov/ports/<port-num>/mcgs/
330
What: /sys/class/infiniband/mlx4_X/iov/ports/<pci-slot-num>/ports/<m>/gid_idx/0
331
What: /sys/class/infiniband/mlx4_X/iov/ports/<pci-slot-num>/ports/<m>/pkey_idx/<n>
332
Date: Aug, 2012
333
KernelVersion: v3.6.15
334
Contact: [email protected]
335
Description:
336
The sysfs iov directory is used to manage and examine the port
337
P_Key and guid paravirtualization. This directory is added only
338
for the master -- slaves do not have it.
340
Under iov/ports, the administrator may examine the gid and P_Key
341
tables as they are present in the device (and as are seen in the
342
"network view" presented to the SM).
344
The "pkeys" and "gids" subdirectories contain one file for each
345
entry in the port's P_Key or GID table respectively. For
346
example, ports/1/pkeys/10 contains the value at index 10 in port
347
1's P_Key table.
349
======================= ==========================================
350
gids/<n>: (RO) The physical port gids n = 0..127
352
admin_guids/<n>: (RW) Allows examining or changing the
353
administrative state of a given GUID
354
n = 0..127
356
pkeys/<n>: (RO) Displays the contents of the physical
357
key table n = 0..126
359
mcgs/: (RO) Multicast group table
361
<m>/gid_idx/0: (RO) Display the GID mapping m = 1..2
363
<m>/pkey_idx/<n>: (RW) Writable except for RoCE pkeys.
364
m = 1..2, n = 0..126
366
Under the iov/<pci slot number>
367
directories, the admin may map the index
368
numbers in the physical tables (as under
369
iov/ports) to the paravirtualized index
370
numbers that guests see.
372
For example, if the administrator, for
373
port 1 on guest 2 maps physical pkey
374
index 10 to virtual index 1, then that
375
guest, whenever it uses its pkey index
376
1, will actually be using the real pkey
377
index 10.
378
======================= ==========================================
381
What: /sys/class/infiniband/mlx4_X/iov/<pci-slot-num>/ports/<m>/smi_enabled
382
What: /sys/class/infiniband/mlx4_X/iov/<pci-slot-num>/ports/<m>/enable_smi_admin
383
Date: May, 2014
384
KernelVersion: v3.15.7
385
Contact: [email protected]
386
Description:
387
Enabling QP0 on VFs for selected VF/port. By default, no VFs are
388
enabled for QP0 operation.
390
================= ==== ===========================================
391
smi_enabled: (RO) Indicates whether smi is currently enabled
392
for the indicated VF/port
394
enable_smi_admin: (RW) Used by the admin to request that smi
395
capability be enabled or disabled for the
396
indicated VF/port. 0 = disable, 1 = enable.
397
================= ==== ===========================================
399
The requested enablement will occur at the next reset of the VF
400
(e.g. driver restart on the VM which owns the VF).
403
sysfs interface for Chelsio T4/T5 RDMA driver (cxgb4)
404
-----------------------------------------------------
406
What: /sys/class/infiniband/cxgb4_X/hw_rev
407
What: /sys/class/infiniband/cxgb4_X/hca_type
408
What: /sys/class/infiniband/cxgb4_X/board_id
409
Date: Apr, 2010
410
KernelVersion: v2.6.35
411
Contact: [email protected]
412
Description:
414
=============== =============================================
415
hw_rev: (RO) Hardware revision number
417
hca_type: (RO) Driver short name. Should normally match
418
the name in its bus driver structure (e.g.
419
pci_driver::name)
421
board_id: (RO) Manufacturing board id. (Vendor + device
422
information)
423
=============== =============================================
426
sysfs interface for Intel IB driver qib
427
---------------------------------------
429
What: /sys/class/infiniband/qibX/version
430
What: /sys/class/infiniband/qibX/hw_rev
431
What: /sys/class/infiniband/qibX/hca_type
432
What: /sys/class/infiniband/qibX/board_id
433
What: /sys/class/infiniband/qibX/boardversion
434
What: /sys/class/infiniband/qibX/nctxts
435
What: /sys/class/infiniband/qibX/localbus_info
436
What: /sys/class/infiniband/qibX/tempsense
437
What: /sys/class/infiniband/qibX/serial
438
What: /sys/class/infiniband/qibX/nfreectxts
439
What: /sys/class/infiniband/qibX/chip_reset
440
Date: May, 2010
441
KernelVersion: v2.6.35
442
Contact: [email protected]
443
Description:
444
=============== ======================================================
445
version: (RO) Display version information of installed software
446
and drivers.
448
hw_rev: (RO) Hardware revision number
450
hca_type: (RO) Host channel adapter type
452
board_id: (RO) Manufacturing board id
454
boardversion: (RO) Current version of the chip architecture
456
nctxts: (RO) Return the number of user ports (contexts)
457
available
459
localbus_info: (RO) Human readable localbus info
461
tempsense: (RO) Display temp sense registers in decimal
463
serial: (RO) Serial number of the HCA
465
nfreectxts: (RO) The number of free user ports (contexts)
466
available.
468
chip_reset: (WO) Reset the chip if possible by writing
469
"reset" to this file. Only allowed if no user
470
contexts are open that use chip resources.
471
=============== ======================================================
474
What: /sys/class/infiniband/qibX/ports/<N>/sl2vl/[0-15]
475
Date: May, 2010
476
KernelVersion: v2.6.35
477
Contact: [email protected]
478
Description:
479
(RO) The directory contains 16 files numbered 0-15 that specify
480
the Service Level (SL). Listing the SL files returns the Virtual
481
Lane (VL) as programmed by the SL.
483
What: /sys/class/infiniband/qibX/ports/<N>/CCMgtA/cc_settings_bin
484
What: /sys/class/infiniband/qibX/ports/<N>/CCMgtA/cc_table_bin
485
Date: May, 2010
486
KernelVersion: v2.6.35
487
Contact: [email protected]
488
Description:
489
Per-port congestion control. Both are binary attributes.
491
=============== ================================================
492
cc_table_bin (RO) Congestion control table size followed by
493
table entries.
495
cc_settings_bin (RO) Congestion settings: port control, control
496
map and an array of 16 entries for the
497
congestion entries - increase, timer, event log
498
trigger threshold and the minimum injection rate
499
delay.
500
=============== ================================================
502
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/loopback
503
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/led_override
504
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/hrtbt_enable
505
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/status
506
What: /sys/class/infiniband/qibX/ports/<N>/linkstate/status_str
507
Date: May, 2010
508
KernelVersion: v2.6.35
509
Contact: [email protected]
510
Description:
511
[to be documented]
513
=============== ===============================================
514
loopback: (WO)
515
led_override: (WO)
516
hrtbt_enable: (RW)
517
status: (RO)
519
status_str: (RO) Displays information about the link state,
520
possible cable/switch problems, and hardware
521
errors. Possible states are- "Initted",
522
"Present", "IB_link_up", "IB_configured" or
523
"Fatal_Hardware_Error".
524
=============== ===============================================
526
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rc_resends
527
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/seq_naks
528
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rdma_seq
529
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rnr_naks
530
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/other_naks
531
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/rc_timeouts
532
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/look_pkts
533
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/pkt_drops
534
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/dma_wait
535
What: /sys/class/infiniband/qibX/ports/<N>/diag_counters/unaligned
536
Date: May, 2010
537
KernelVersion: v2.6.35
538
Contact: [email protected]
539
Description:
540
[to be documented]
543
sysfs interface for Mellanox Connect-IB HCA driver mlx5
544
-------------------------------------------------------
546
What: /sys/class/infiniband/mlx5_X/hw_rev
547
What: /sys/class/infiniband/mlx5_X/hca_type
548
What: /sys/class/infiniband/mlx5_X/reg_pages
549
What: /sys/class/infiniband/mlx5_X/fw_pages
550
Date: Jul, 2013
551
KernelVersion: v3.11
552
Contact: [email protected]
553
Description:
554
[to be documented]
557
sysfs interface for Cisco VIC (usNIC) Verbs Driver
558
--------------------------------------------------
560
What: /sys/class/infiniband/usnic_X/board_id
561
What: /sys/class/infiniband/usnic_X/config
562
What: /sys/class/infiniband/usnic_X/qp_per_vf
563
What: /sys/class/infiniband/usnic_X/max_vf
564
What: /sys/class/infiniband/usnic_X/cq_per_vf
565
What: /sys/class/infiniband/usnic_X/iface
566
Date: Sep, 2013
567
KernelVersion: v3.14
568
Contact: Christian Benvenuti <[email protected]>,
569
Dave Goodell <[email protected]>,
571
Description:
573
=============== ===============================================
574
board_id: (RO) Manufacturing board id
576
config: (RO) Report the configuration for this PF
578
qp_per_vf: (RO) Queue pairs per virtual function.
580
max_vf: (RO) Max virtual functions
582
cq_per_vf: (RO) Completion queue per virtual function
584
iface: (RO) Shows which network interface this usNIC
585
entry is associated to (visible with ifconfig).
586
=============== ===============================================
588
What: /sys/class/infiniband/usnic_X/qpn/summary
589
What: /sys/class/infiniband/usnic_X/qpn/context
590
Date: Sep, 2013
591
KernelVersion: v3.14
592
Contact: Christian Benvenuti <[email protected]>,
593
Dave Goodell <[email protected]>,
595
Description:
596
[to be documented]
599
sysfs interface for Emulex RoCE HCA Driver
600
------------------------------------------
602
What: /sys/class/infiniband/ocrdmaX/hw_rev
603
Date: Feb, 2014
604
KernelVersion: v3.14
605
Description:
606
hw_rev: (RO) Hardware revision number
608
What: /sys/class/infiniband/ocrdmaX/hca_type
609
Date: Jun, 2014
610
KernelVersion: v3.16
611
Contact: [email protected]
612
Description:
613
hca_type: (RO) Display FW version
616
sysfs interface for Intel Omni-Path driver (HFI1)
617
-------------------------------------------------
619
What: /sys/class/infiniband/hfi1_X/hw_rev
620
What: /sys/class/infiniband/hfi1_X/board_id
621
What: /sys/class/infiniband/hfi1_X/nctxts
622
What: /sys/class/infiniband/hfi1_X/serial
623
What: /sys/class/infiniband/hfi1_X/chip_reset
624
What: /sys/class/infiniband/hfi1_X/boardversion
625
What: /sys/class/infiniband/hfi1_X/nfreectxts
626
What: /sys/class/infiniband/hfi1_X/tempsense
627
Date: May, 2016
628
KernelVersion: v4.6
629
Contact: [email protected]
630
Description:
631
=============== =============================================
632
hw_rev: (RO) Hardware revision number
634
board_id: (RO) Manufacturing board id
636
nctxts: (RO) Total contexts available.
638
serial: (RO) Board serial number
640
chip_reset: (WO) Write "reset" to this file to reset the
641
chip if possible. Only allowed if no user
642
contexts are open that use chip resources.
644
boardversion: (RO) Human readable board info
646
nfreectxts: (RO) The number of free user ports (contexts)
647
available.
649
tempsense: (RO) Thermal sense information
650
=============== =============================================
653
What: /sys/class/infiniband/hfi1_X/ports/<N>/CCMgtA/cc_settings_bin
654
What: /sys/class/infiniband/hfi1_X/ports/<N>/CCMgtA/cc_table_bin
655
What: /sys/class/infiniband/hfi1_X/ports/<N>/CCMgtA/cc_prescan
656
Date: May, 2016
657
KernelVersion: v4.6
658
Contact: [email protected]
659
Description:
660
Per-port congestion control.
662
=============== ================================================
663
cc_table_bin (RO) CCA tables used by PSM2 Congestion control
664
table size followed by table entries. Binary
665
attribute.
667
cc_settings_bin (RO) Congestion settings: port control, control
668
map and an array of 16 entries for the
669
congestion entries - increase, timer, event log
670
trigger threshold and the minimum injection rate
671
delay. Binary attribute.
673
cc_prescan (RW) enable prescanning for faster BECN
674
response. Write "on" to enable and "off" to
675
disable.
676
=============== ================================================
678
What: /sys/class/infiniband/hfi1_X/ports/<N>/sc2vl/[0-31]
679
What: /sys/class/infiniband/hfi1_X/ports/<N>/sl2sc/[0-31]
680
What: /sys/class/infiniband/hfi1_X/ports/<N>/vl2mtu/[0-15]
681
Date: May, 2016
682
KernelVersion: v4.6
683
Contact: [email protected]
684
Description:
685
=============== ===================================================
686
sc2vl/: (RO) 32 files (0 - 31) used to translate sl->vl
688
sl2sc/: (RO) 32 files (0 - 31) used to translate sl->sc
690
vl2mtu/: (RO) 16 files (0 - 15) used to determine MTU for vl
691
=============== ===================================================
694
What: /sys/class/infiniband/hfi1_X/sdma_<N>/cpu_list
695
What: /sys/class/infiniband/hfi1_X/sdma_<N>/vl
696
Date: Sept, 2016
697
KernelVersion: v4.8
698
Contact: [email protected]
699
Description:
700
sdma<N>/ contains one directory per sdma engine (0 - 15)
702
=============== ==============================================
703
cpu_list: (RW) List of cpus for user-process to sdma
704
engine assignment.
706
vl: (RO) Displays the virtual lane (vl) the sdma
707
engine maps to.
708
=============== ==============================================
710
This interface gives the user control on the affinity settings
711
for the device. As an example, to set an sdma engine irq
712
affinity and thread affinity of a user processes to use the
713
sdma engine, which is "near" in terms of NUMA configuration, or
714
physical cpu location, the user will do::
716
echo "3" > /proc/irq/<N>/smp_affinity_list
717
echo "4-7" > /sys/devices/.../sdma3/cpu_list
718
cat /sys/devices/.../sdma3/vl
719
0
720
echo "8" > /proc/irq/<M>/smp_affinity_list
721
echo "9-12" > /sys/devices/.../sdma4/cpu_list
722
cat /sys/devices/.../sdma4/vl
723
1
725
to make sure that when a process runs on cpus 4,5,6, or 7, and
726
uses vl=0, then sdma engine 3 is selected by the driver, and
727
also the interrupt of the sdma engine 3 is steered to cpu 3.
728
Similarly, when a process runs on cpus 9,10,11, or 12 and sets
729
vl=1, then engine 4 will be selected and the irq of the sdma
730
engine 4 is steered to cpu 8. This assumes that in the above N
731
is the irq number of "sdma3", and M is irq number of "sdma4" in
732
the /proc/interrupts file.
734
sysfs interface for QLogic qedr NIC Driver
735
------------------------------------------
737
What: /sys/class/infiniband/qedrX/hw_rev
738
What: /sys/class/infiniband/qedrX/hca_type
739
Date: Oct, 2016
740
KernelVersion: v4.10
741
Contact: [email protected]
742
Description:
744
=============== ==== ========================
745
hw_rev: (RO) Hardware revision number
747
hca_type: (RO) Display HCA type
748
=============== ==== ========================
751
sysfs interface for VMware Paravirtual RDMA driver
752
--------------------------------------------------
754
What: /sys/class/infiniband/vmw_pvrdmaX/hw_rev
755
What: /sys/class/infiniband/vmw_pvrdmaX/hca_type
756
What: /sys/class/infiniband/vmw_pvrdmaX/board_id
757
Date: Oct, 2016
758
KernelVersion: v4.10
759
Contact: [email protected]
760
Description:
762
=============== ==== =====================================
763
hw_rev: (RO) Hardware revision number
765
hca_type: (RO) Host channel adapter type
767
board_id: (RO) Display PVRDMA manufacturing board ID
768
=============== ==== =====================================
771
sysfs interface for Broadcom NetXtreme-E RoCE driver
772
----------------------------------------------------
774
What: /sys/class/infiniband/bnxt_reX/hw_rev
775
What: /sys/class/infiniband/bnxt_reX/hca_type
776
Date: Feb, 2017
777
KernelVersion: v4.11
778
Contact: [email protected]
779
Description:
780
=============== ==== =========================
781
hw_rev: (RO) Hardware revision number
783
hca_type: (RO) Host channel adapter type
784
=============== ==== =========================
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
모든 InfiniBand device의 공통 attribute
1-36공통 interface의 contact는 [email protected]다.
| 경로 | 도입·접근 | 의미 |
|---|---|---|
| /sys/class/infiniband/<device>/node_type | 2005년 4월 · kernel 2.6.12 · read-only | Node type이다. CA, RNIC, usNIC, usNIC UDP, switch 또는 router 가운데 하나다. |
| /sys/class/infiniband/<device>/node_guid | 2005년 4월 · kernel 2.6.12 · read-only | Node GUID다. |
| /sys/class/infiniband/<device>/sys_image_guid | 2005년 4월 · kernel 2.6.12 · read-only | System image GUID다. |
| /sys/class/infiniband/<device>/node_desc | 2006년 2월 · kernel 2.6.17 · read/write | Hostname 같은 정보로 node description을 갱신해 IB network management software의 view를 실제 system과 연결한다. |
| /sys/class/infiniband/<device>/fw_ver | 2016년 6월 · kernel 4.10 · read-only | Firmware version을 표시한다. |
Port identity, state와 capability
39-82Port 기본 attribute는 2005년 4월 kernel 2.6.12에 도입됐고 모두 read-only다.
| attribute | 의미 |
|---|---|
| lid | Port LID |
| rate | active width와 active speed의 곱인 port data rate |
| lid_mask_count | Port LID mask count |
| sm_sl | Port subnet의 subnet manager SL |
| sm_lid | Port subnet의 subnet manager LID |
| state | DOWN, INIT, ARMED, ACTIVE 또는 ACTIVE_DEFER인 port state |
| phys_state | Sleep, Polling, LinkUp 등 port physical state |
| cap_mask | Port capability mask다. IsCommunicationManagementSupported는 CM module이 load되면 설정되고, IsSM은 issmN file을 open하면 설정되는 두 writable bit다. |
/sys/class/infiniband/<device>/ports/<port-num>/link_layer는 2010년 10월 kernel 2.6.37에 추가된 read-only attribute로 link layer가 InfiniBand인지 Ethernet인지 표시한다.
Port error counter
85-146아래 read-only counter는 2005년 4월 kernel 2.6.12에 도입됐다.
| counter | 의미 |
|---|---|
| symbol_error | 하나 이상의 physical lane에서 감지한 minor link error의 전체 수 |
| port_rcv_errors | Port가 수신한 error 포함 packet의 전체 수 |
| port_rcv_remote_physical_errors | Port가 수신한 EBP delimiter 표시 packet의 전체 수 |
| port_rcv_switch_relay_errors | Switch relay가 forward할 수 없어 버린 수신 packet의 전체 수 |
| link_error_recovery | Port Training state machine이 link error recovery를 성공적으로 완료한 횟수 |
| port_xmit_constraint_errors | Outbound raw filtering 또는 partition/IP version check 실패로 switch physical port에서 전송하지 않은 packet의 전체 수 |
| port_rcv_contraint_errors | Inbound raw filtering 또는 partition/IP version check 실패로 switch physical port가 버린 수신 packet의 전체 수. Path에는 원문의 contraint 철자가 사용된다. |
| local_link_integrity_errors | Local physical error 수가 LocalPhyErrors threshold를 넘은 횟수 |
| excessive_buffer_overrun_errors | Input buffer overrun counter다. SM이나 user의 port misconfiguration, hardware 문제 또는 매우 나쁜 link signal integrity를 나타낼 수 있다. |
Port data와 기타 counter
148-196| counter | 의미 |
|---|---|
| port_xmit_data | 모든 VL에서 전송한 data octet 수를 4 lane으로 나눈 64-bit 값 |
| port_rcv_data | 모든 VL에서 수신한 data octet 수를 4 lane으로 나눈 64-bit 값 |
| port_xmit_packets | 이 port가 모든 VL에서 전송한 전체 packet 수. Error packet을 포함할 수 있는 64-bit counter다. |
| port_rcv_packets | 수신한 전체 packet 수. Error 포함 packet을 포함할 수 있는 64-bit counter다. |
| unicast_rcv_packets | Error 포함 unicast packet을 포함한 전체 수신 수 |
| unicast_xmit_packets | 모든 VL에서 전송한 전체 unicast packet 수. Error packet을 포함할 수 있다. |
| multicast_rcv_packets | Error 포함 multicast packet을 포함한 전체 수신 수 |
| multicast_xmit_packets | 모든 VL에서 전송한 전체 multicast packet 수. Error packet을 포함할 수 있다. |
| link_downed | Port Training state machine이 link error recovery에 실패하고 link를 down시킨 전체 횟수 |
| port_xmit_discards | Port down 또는 congestion 때문에 port가 버린 outbound packet 전체 수 |
| VL15_dropped | Buffer 부족 같은 port resource 제한 때문에 버린 incoming VL15 packet 수 |
| port_xmit_wait | 전송할 data가 있었지만 credit 부족 또는 arbitration 부재로 tick 전체 동안 아무 data도 보내지 못한 tick 수 |
각 file은 InfiniBand Architecture Specification의 port Performance Management PortCounters attribute에 있는 대응 값을 담는다.
Hardware counter cache와 RoCE GID
199-2322016년 5월 kernel 4.6에 추가된 optional hw_counters subdirectory는 parent device, port 또는 양쪽 모두에 있을 수 있다. Hardware가 제공하는 counter 목록은 일반 counters와 일부 겹칠 수 있으나 다른 counter도 흔히 포함한다.
device 또는 port의 hw_counters/lifespan은 counter에 접근할 때 core가 값을 갱신하는 주기를 millisecond로 설정한다. 접근하지 않으면 갱신하지 않는다. Driver가 다르게 정하지 않으면 기본값은 10ms이고, 0부터 10000 사이 값을 써서 갱신 간격을 바꿀 수 있다.
| 경로 | 도입 | 의미 |
|---|---|---|
| .../gid_attrs/ndevs/<gid-index> | 2015년 11월 29일 · kernel 4.4.0 | 해당 GID에 연결된 net-device 이름을 gid-index 위치에 제공한다. |
| .../gid_attrs/types/<gid-index> | 2015년 11월 29일 · kernel 4.4.0 | GID의 RoCE type을 제공한다. IB와 RoCE v1 기반 GID는 IB/RoCE v1, RoCE v2 기반 GID는 RoCE v2다. |
MAD, issm와 uverbs ABI
235-284각 InfiniBand device port에는 umad device와 issm device가 하나씩 연결된다. 2-port HCA는 각각 두 개를 가지며 switch는 port 0에 대해 각 type 하나를 갖는다.
| 경로 | 도입 | 의미 |
|---|---|---|
| /sys/class/infiniband_mad/umad<N>/ibdev 및 port | 2005년 4월 · kernel 2.6.12 | ibdev는 IB device 이름, port는 port number를 표시하는 read-only attribute다. |
| /sys/class/infiniband_mad/issm<N>/ibdev 및 port | 2005년 4월 · kernel 2.6.12 | ibdev는 IB device 이름, port는 port number를 표시하는 read-only attribute다. |
| /sys/class/infiniband_mad/abi_version | 2005년 4월 · kernel 2.6.12 | umad 또는 issm device의 user space ABI compatibility를 깨는 변경이 생기면 증가하는 read-only 값이다. |
| /sys/class/infiniband_verbs/uverbs<N>/ibdev | 2005년 9월 · kernel 2.6.14 | IB device 이름을 표시한다. |
| /sys/class/infiniband_verbs/uverbs<N>/abi_version | 2005년 9월 · kernel 2.6.14 | IB device별 interface의 ABI version을 표시한다. |
| /sys/class/infiniband_verbs/abi_version | 2005년 9월 · kernel 2.6.14 | uverbs device의 user space ABI compatibility를 깨는 변경이 생기면 증가하는 read-only 값이다. |
Mellanox mthca와 mlx4 HCA identity
287-323| Driver·도입 | attribute | 의미 |
|---|---|---|
| mthca · 2005년 4월 · kernel 2.6.12 | hw_rev | Hardware revision number |
| mthca | hca_type | MT23108, MT25208(MT23108 compatibility mode), MT25208 또는 MT25204 HCA type |
| mthca | board_id | Manufacturing board ID |
| mlx4 · 2007년 9월 · kernel 2.6.24 | hw_rev | Hardware revision number |
| mlx4 | hca_type | Host channel adapter type |
| mlx4 | board_id | Manufacturing board ID |
모두 /sys/class/infiniband/mthcaX/ 또는 /sys/class/infiniband/mlx4_X/ 아래의 read-only attribute다.
mlx4 SR-IOV GID와 P_Key paravirtualization
326-4002012년 8월 kernel 3.6.15에 도입된 iov directory는 port P_Key와 GUID paravirtualization을 관리·검사한다. Master에만 생기며 slave에는 없다. Administrator는 iov/ports에서 device의 실제 GID·P_Key table과 SM에 보이는 network view를 확인한다.
| 경로 | 접근과 범위 |
|---|---|
| iov/ports/<port-num>/gids/<n> | read-only physical port GID, n=0..127 |
| iov/ports/<port-num>/admin_guids/<n> | GUID administrative state 조회·변경, read/write, n=0..127 |
| iov/ports/<port-num>/pkeys/<n> | read-only physical P_Key table, n=0..126 |
| iov/ports/<port-num>/mcgs/ | read-only multicast group table |
| iov/ports/<pci-slot-num>/ports/<m>/gid_idx/0 | read-only GID mapping, m=1..2 |
| iov/ports/<pci-slot-num>/ports/<m>/pkey_idx/<n> | RoCE P_Key를 제외하고 read/write, m=1..2, n=0..126 |
pkeys와 gids subdirectory는 physical table entry마다 file 하나를 둔다. 예를 들어 ports/1/pkeys/10은 port 1 P_Key table의 index 10 값이다. iov/<pci slot number> 아래에서 physical table index를 guest가 보는 paravirtualized index에 mapping한다. Guest 2의 port 1에서 physical P_Key index 10을 virtual index 1에 mapping하면 guest가 index 1을 쓸 때 실제 index 10을 사용한다.
2014년 5월 kernel 3.15.7에 추가된 smi_enabled는 해당 VF/port의 SMI 활성 상태를 읽고, enable_smi_admin은 administrator가 0으로 비활성화 또는 1로 활성화를 요청한다. 기본적으로 어떤 VF도 QP0 operation이 허용되지 않는다. 요청은 다음 VF reset, 예를 들어 VF를 소유한 VM의 driver restart 때 적용된다.
Chelsio cxgb4와 Intel qib
403-540cxgb4 attribute는 2010년 4월 kernel 2.6.35에 도입된 read-only 값이다. hw_rev는 hardware revision, hca_type은 보통 pci_driver::name 같은 bus driver structure 이름과 일치하는 driver short name, board_id는 vendor와 device 정보를 포함한 manufacturing board ID다.
qib attribute는 2010년 5월 kernel 2.6.35에 도입됐다.
| qib attribute | 접근 | 의미 |
|---|---|---|
| version | read-only | 설치된 software와 driver version |
| hw_rev | read-only | Hardware revision number |
| hca_type | read-only | Host channel adapter type |
| board_id | read-only | Manufacturing board ID |
| boardversion | read-only | 현재 chip architecture version |
| nctxts | read-only | 사용 가능한 user port(context) 전체 수 |
| localbus_info | read-only | 사람이 읽을 수 있는 local bus 정보 |
| tempsense | read-only | Temperature sense register의 10진수 값 |
| serial | read-only | HCA serial number |
| nfreectxts | read-only | 사용 가능한 free user context 수 |
| chip_reset | write-only | reset 문자열을 써서 가능한 경우 chip reset. Chip resource를 쓰는 open user context가 없어야 한다. |
qib ports/<N>/sl2vl/[0-15]는 0부터 15까지 16개 read-only file이며 각 Service Level에 programming된 Virtual Lane을 반환한다.
| qib congestion attribute | 의미 |
|---|---|
| CCMgtA/cc_table_bin | read-only binary. Congestion control table 크기 뒤에 table entry가 이어진다. |
| CCMgtA/cc_settings_bin | read-only binary. Port control, control map과 increase, timer, event log trigger threshold, minimum injection rate delay를 담은 16개 congestion entry 배열이다. |
qib linkstate의 loopback(write-only), led_override(write-only), hrtbt_enable(read/write), status(read-only)는 원문에 추가 설명이 없다. status_str은 link state, cable·switch 문제와 hardware error 정보를 표시하며 가능한 상태는 Initted, Present, IB_link_up, IB_configured, Fatal_Hardware_Error다.
qib diag_counters의 rc_resends, seq_naks, rdma_seq, rnr_naks, other_naks, rc_timeouts, look_pkts, pkt_drops, dma_wait, unaligned는 원문에서 [to be documented]로 남아 있다.
mlx5, usNIC와 ocrdma
543-613mlx5의 hw_rev, hca_type, reg_pages, fw_pages는 2013년 7월 kernel 3.11에 도입됐으나 원문에는 [to be documented]만 적혀 있다.
Cisco usNIC attribute는 2013년 9월 kernel 3.14에 도입됐고 담당자는 Christian Benvenuti, Dave Goodell과 linux-rdma mailing list다.
| usNIC attribute | 의미 |
|---|---|
| board_id | Manufacturing board ID |
| config | 이 PF의 configuration |
| qp_per_vf | Virtual function당 queue pair 수 |
| max_vf | 최대 virtual function 수 |
| cq_per_vf | Virtual function당 completion queue 수 |
| iface | 이 usNIC entry와 연결된 ifconfig에서 보이는 network interface |
usnic_X/qpn/summary와 qpn/context는 원문에서 [to be documented]로 남아 있다.
Emulex ocrdma의 hw_rev는 2014년 2월 kernel 3.14에 도입된 read-only hardware revision number다. hca_type은 2014년 6월 kernel 3.16에 도입됐으며 원문 설명은 read-only FW version 표시라고 되어 있다.
Intel Omni-Path HFI1
616-691HFI1 기본 attribute는 2016년 5월 kernel 4.6에 도입됐다.
| attribute | 접근 | 의미 |
|---|---|---|
| hw_rev | read-only | Hardware revision number |
| board_id | read-only | Manufacturing board ID |
| nctxts | read-only | 사용 가능한 전체 context 수 |
| serial | read-only | Board serial number |
| chip_reset | write-only | reset을 써서 가능한 경우 chip을 reset한다. Chip resource를 쓰는 open user context가 없어야 한다. |
| boardversion | read-only | 사람이 읽을 수 있는 board 정보 |
| nfreectxts | read-only | 사용 가능한 free user context 수 |
| tempsense | read-only | Thermal sense 정보 |
| Per-port congestion attribute | 의미 |
|---|---|
| CCMgtA/cc_table_bin | PSM2 congestion control이 쓰는 CCA table 크기와 entry를 담은 read-only binary attribute |
| CCMgtA/cc_settings_bin | Port control, control map, increase·timer·event log threshold·minimum injection rate delay의 16개 entry를 담은 read-only binary attribute |
| CCMgtA/cc_prescan | 더 빠른 BECN response를 위한 prescan을 on으로 켜고 off로 끄는 read/write attribute |
| Mapping directory | 의미 |
|---|---|
| sc2vl/[0-31] | SL에서 VL로 변환하는 32개 read-only file |
| sl2sc/[0-31] | SL에서 SC로 변환하는 32개 read-only file |
| vl2mtu/[0-15] | VL의 MTU를 정하는 16개 read-only file |
HFI1 SDMA engine affinity
694-7322016년 9월 kernel 4.8에 도입됐다. sdma_<N>/은 0부터 15까지 SDMA engine마다 directory 하나를 둔다. cpu_list는 user process를 SDMA engine에 할당할 CPU 목록을 설정하는 read/write attribute고, vl은 engine이 mapping되는 virtual lane을 표시하는 read-only attribute다.
NUMA 구성이나 physical CPU 위치상 가까운 SDMA engine을 사용하도록 user process thread affinity와 engine IRQ affinity를 함께 제어할 수 있다.
echo "3" > /proc/irq/<N>/smp_affinity_list
echo "4-7" > /sys/devices/.../sdma3/cpu_list
cat /sys/devices/.../sdma3/vl
0
echo "8" > /proc/irq/<M>/smp_affinity_list
echo "9-12" > /sys/devices/.../sdma4/cpu_list
cat /sys/devices/.../sdma4/vl
1CPU 4–7에서 실행하며 VL 0을 쓰는 process에는 engine 3을 선택하고 engine 3 interrupt는 CPU 3으로 보낸다. CPU 9–12에서 VL 1을 쓰는 process에는 engine 4를 선택하고 interrupt는 CPU 8로 보낸다. N과 M은 /proc/interrupts에 나온 sdma3과 sdma4의 IRQ number라고 가정한다.
qedr, VMware PVRDMA와 Broadcom bnxt_re
734-784| Driver·도입 | attribute | 의미 |
|---|---|---|
| QLogic qedr · 2016년 10월 · kernel 4.10 | hw_rev | read-only hardware revision number |
| QLogic qedr | hca_type | read-only HCA type |
| VMware PVRDMA · 2016년 10월 · kernel 4.10 | hw_rev | read-only hardware revision number |
| VMware PVRDMA | hca_type | read-only host channel adapter type |
| VMware PVRDMA | board_id | read-only PVRDMA manufacturing board ID |
| Broadcom bnxt_re · 2017년 2월 · kernel 4.11 | hw_rev | read-only hardware revision number |
| Broadcom bnxt_re | hca_type | read-only host channel adapter type |
이 driver별 attribute의 contact는 [email protected]다.
RDMA core device와 port
sysfs-class-infiniband:1-82Node GUID·description·firmware와 port LID·rate·state·capability·link layer를 다룹니다.