요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
1
.. SPDX-License-Identifier: GPL-2.0
3
===================================
4
Linux Ethernet Bonding Driver HOWTO
5
===================================
7
Latest update: 27 April 2011
9
Initial release: Thomas Davis <tadavis at lbl.gov>
11
Corrections, HA extensions: 2000/10/03-15:
13
- Willy Tarreau <willy at meta-x.org>
14
- Constantine Gavrilov <const-g at xpert.com>
15
- Chad N. Tindel <ctindel at ieee dot org>
16
- Janice Girouard <girouard at us dot ibm dot com>
17
- Jay Vosburgh <fubar at us dot ibm dot com>
19
Reorganized and updated Feb 2005 by Jay Vosburgh
20
Added Sysfs information: 2006/04/24
22
- Mitch Williams <mitch.a.williams at intel.com>
24
Introduction
25
============
27
The Linux bonding driver provides a method for aggregating
28
multiple network interfaces into a single logical "bonded" interface.
29
The behavior of the bonded interfaces depends upon the mode; generally
30
speaking, modes provide either hot standby or load balancing services.
31
Additionally, link integrity monitoring may be performed.
33
The bonding driver originally came from Donald Becker's
34
beowulf patches for kernel 2.0. It has changed quite a bit since, and
35
the original tools from extreme-linux and beowulf sites will not work
36
with this version of the driver.
38
For new versions of the driver, updated userspace tools, and
39
who to ask for help, please follow the links at the end of this file.
41
.. Table of Contents
43
1. Bonding Driver Installation
45
2. Bonding Driver Options
47
3. Configuring Bonding Devices
48
3.1 Configuration with Sysconfig Support
49
3.1.1 Using DHCP with Sysconfig
50
3.1.2 Configuring Multiple Bonds with Sysconfig
51
3.2 Configuration with Initscripts Support
52
3.2.1 Using DHCP with Initscripts
53
3.2.2 Configuring Multiple Bonds with Initscripts
54
3.3 Configuring Bonding Manually with Ifenslave
55
3.3.1 Configuring Multiple Bonds Manually
56
3.4 Configuring Bonding Manually via Sysfs
57
3.5 Configuration with Interfaces Support
58
3.6 Overriding Configuration for Special Cases
59
3.7 Configuring LACP for 802.3ad mode in a more secure way
61
4. Querying Bonding Configuration
62
4.1 Bonding Configuration
63
4.2 Network Configuration
65
5. Switch Configuration
67
6. 802.1q VLAN Support
69
7. Link Monitoring
70
7.1 ARP Monitor Operation
71
7.2 Configuring Multiple ARP Targets
72
7.3 MII Monitor Operation
74
8. Potential Trouble Sources
75
8.1 Adventures in Routing
76
8.2 Ethernet Device Renaming
77
8.3 Painfully Slow Or No Failed Link Detection By Miimon
79
9. SNMP agents
81
10. Promiscuous mode
83
11. Configuring Bonding for High Availability
84
11.1 High Availability in a Single Switch Topology
85
11.2 High Availability in a Multiple Switch Topology
86
11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
87
11.2.2 HA Link Monitoring for Multiple Switch Topology
89
12. Configuring Bonding for Maximum Throughput
90
12.1 Maximum Throughput in a Single Switch Topology
91
12.1.1 MT Bonding Mode Selection for Single Switch Topology
92
12.1.2 MT Link Monitoring for Single Switch Topology
93
12.2 Maximum Throughput in a Multiple Switch Topology
94
12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
95
12.2.2 MT Link Monitoring for Multiple Switch Topology
97
13. Switch Behavior Issues
98
13.1 Link Establishment and Failover Delays
99
13.2 Duplicated Incoming Packets
101
14. Hardware Specific Considerations
102
14.1 IBM BladeCenter
104
15. Frequently Asked Questions
106
16. Resources and Links
109
1. Bonding Driver Installation
110
==============================
112
Most popular distro kernels ship with the bonding driver
113
already available as a module. If your distro does not, or you
114
have need to compile bonding from source (e.g., configuring and
115
installing a mainline kernel from kernel.org), you'll need to perform
116
the following steps:
118
1.1 Configure and build the kernel with bonding
119
-----------------------------------------------
121
The current version of the bonding driver is available in the
122
drivers/net/bonding subdirectory of the most recent kernel source
123
(which is available on http://kernel.org). Most users "rolling their
124
own" will want to use the most recent kernel from kernel.org.
126
Configure kernel with "make menuconfig" (or "make xconfig" or
127
"make config"), then select "Bonding driver support" in the "Network
128
device support" section. It is recommended that you configure the
129
driver as module since it is currently the only way to pass parameters
130
to the driver or configure more than one bonding device.
132
Build and install the new kernel and modules.
134
1.2 Bonding Control Utility
135
---------------------------
137
It is recommended to configure bonding via iproute2 (netlink)
138
or sysfs, the old ifenslave control utility is obsolete.
140
2. Bonding Driver Options
141
=========================
143
Options for the bonding driver are supplied as parameters to the
144
bonding module at load time, or are specified via sysfs.
146
Module options may be given as command line arguments to the
147
insmod or modprobe command, but are usually specified in either the
148
``/etc/modprobe.d/*.conf`` configuration files, or in a distro-specific
149
configuration file (some of which are detailed in the next section).
151
Details on bonding support for sysfs is provided in the
152
"Configuring Bonding Manually via Sysfs" section, below.
154
The available bonding driver parameters are listed below. If a
155
parameter is not specified the default value is used. When initially
156
configuring a bond, it is recommended "tail -f /var/log/messages" be
157
run in a separate window to watch for bonding driver error messages.
159
It is critical that either the miimon or arp_interval and
160
arp_ip_target parameters be specified, otherwise serious network
161
degradation will occur during link failures. Very few devices do not
162
support at least miimon, so there is really no reason not to use it.
164
Options with textual values will accept either the text name
165
or, for backwards compatibility, the option value. E.g.,
166
"mode=802.3ad" and "mode=4" set the same mode.
168
The parameters are as follows:
170
active_slave
172
Specifies the new active slave for modes that support it
173
(active-backup, balance-alb and balance-tlb). Possible values
174
are the name of any currently enslaved interface, or an empty
175
string. If a name is given, the slave and its link must be up in order
176
to be selected as the new active slave. If an empty string is
177
specified, the current active slave is cleared, and a new active
178
slave is selected automatically.
180
Note that this is only available through the sysfs interface. No module
181
parameter by this name exists.
183
The normal value of this option is the name of the currently
184
active slave, or the empty string if there is no active slave or
185
the current mode does not use an active slave.
187
ad_actor_sys_prio
189
In an AD system, this specifies the system priority. The allowed range
190
is 1 - 65535. If the value is not specified, it takes 65535 as the
191
default value.
193
This parameter has effect only in 802.3ad mode and is available through
194
SysFs interface.
196
actor_port_prio
198
In an AD system, this specifies the port priority. The allowed range
199
is 1 - 65535. If the value is not specified, it takes 255 as the
200
default value.
202
This parameter has effect only in 802.3ad mode and is available through
203
netlink interface.
205
ad_actor_system
207
In an AD system, this specifies the mac-address for the actor in
208
protocol packet exchanges (LACPDUs). The value cannot be a multicast
209
address. If the all-zeroes MAC is specified, bonding will internally
210
use the MAC of the bond itself. It is preferred to have the
211
local-admin bit set for this mac but driver does not enforce it. If
212
the value is not given then system defaults to using the masters'
213
mac address as actors' system address.
215
This parameter has effect only in 802.3ad mode and is available through
216
SysFs interface.
218
ad_select
220
Specifies the 802.3ad aggregation selection logic to use. The
221
possible values and their effects are:
223
stable or 0
225
The active aggregator is chosen by largest aggregate
226
bandwidth.
228
Reselection of the active aggregator occurs only when all
229
slaves of the active aggregator are down or the active
230
aggregator has no slaves.
232
This is the default value.
234
bandwidth or 1
236
The active aggregator is chosen by largest aggregate
237
bandwidth. Reselection occurs if:
239
- A slave is added to or removed from the bond
241
- Any slave's link state changes
243
- Any slave's 802.3ad association state changes
245
- The bond's administrative state changes to up
247
count or 2
249
The active aggregator is chosen by the largest number of
250
ports (slaves). Reselection occurs as described under the
251
"bandwidth" setting, above.
253
actor_port_prio or 3
255
The active aggregator is chosen by the highest total sum of
256
actor port priorities across its active ports. Note this
257
priority is actor_port_prio, not per port prio, which is
258
used for primary reselect.
260
The bandwidth, count and actor_port_prio selection policies permit
261
failover of 802.3ad aggregations when partial failure of the active
262
aggregator occurs. This keeps the aggregator with the highest
263
availability (either in bandwidth, number of ports, or total value
264
of port priorities) active at all times.
266
This option was added in bonding version 3.4.0.
268
ad_user_port_key
270
In an AD system, the port-key has three parts as shown below -
272
===== ============
273
Bits Use
274
===== ============
275
00 Duplex
276
01-05 Speed
277
06-15 User-defined
278
===== ============
280
This defines the upper 10 bits of the port key. The values can be
281
from 0 - 1023. If not given, the system defaults to 0.
283
This parameter has effect only in 802.3ad mode and is available through
284
SysFs interface.
286
all_slaves_active
288
Specifies that duplicate frames (received on inactive ports) should be
289
dropped (0) or delivered (1).
291
Normally, bonding will drop duplicate frames (received on inactive
292
ports), which is desirable for most users. But there are some times
293
it is nice to allow duplicate frames to be delivered.
295
The default value is 0 (drop duplicate frames received on inactive
296
ports).
298
arp_interval
300
Specifies the ARP link monitoring frequency in milliseconds.
302
The ARP monitor works by periodically checking the slave
303
devices to determine whether they have sent or received
304
traffic recently (the precise criteria depends upon the
305
bonding mode, and the state of the slave). Regular traffic is
306
generated via ARP probes issued for the addresses specified by
307
the arp_ip_target option.
309
This behavior can be modified by the arp_validate option,
310
below.
312
If ARP monitoring is used in an etherchannel compatible mode
313
(modes 0 and 2), the switch should be configured in a mode
314
that evenly distributes packets across all links. If the
315
switch is configured to distribute the packets in an XOR
316
fashion, all replies from the ARP targets will be received on
317
the same link which could cause the other team members to
318
fail. ARP monitoring should not be used in conjunction with
319
miimon. A value of 0 disables ARP monitoring. The default
320
value is 0.
322
arp_ip_target
324
Specifies the IP addresses to use as ARP monitoring peers when
325
arp_interval is > 0. These are the targets of the ARP request
326
sent to determine the health of the link to the targets.
327
Specify these values in ddd.ddd.ddd.ddd format. Multiple IP
328
addresses must be separated by a comma. At least one IP
329
address must be given for ARP monitoring to function. The
330
maximum number of targets that can be specified is 16. The
331
default value is no IP addresses.
333
ns_ip6_target
335
Specifies the IPv6 addresses to use as IPv6 monitoring peers when
336
arp_interval is > 0. These are the targets of the NS request
337
sent to determine the health of the link to the targets.
338
Specify these values in ffff:ffff::ffff:ffff format. Multiple IPv6
339
addresses must be separated by a comma. At least one IPv6
340
address must be given for NS/NA monitoring to function. The
341
maximum number of targets that can be specified is 16. The
342
default value is no IPv6 addresses.
344
arp_validate
346
Specifies whether or not ARP probes and replies should be
347
validated in any mode that supports arp monitoring, or whether
348
non-ARP traffic should be filtered (disregarded) for link
349
monitoring purposes.
351
Possible values are:
353
none or 0
355
No validation or filtering is performed.
357
active or 1
359
Validation is performed only for the active slave.
361
backup or 2
363
Validation is performed only for backup slaves.
365
all or 3
367
Validation is performed for all slaves.
369
filter or 4
371
Filtering is applied to all slaves. No validation is
372
performed.
374
filter_active or 5
376
Filtering is applied to all slaves, validation is performed
377
only for the active slave.
379
filter_backup or 6
381
Filtering is applied to all slaves, validation is performed
382
only for backup slaves.
384
Validation:
386
Enabling validation causes the ARP monitor to examine the incoming
387
ARP requests and replies, and only consider a slave to be up if it
388
is receiving the appropriate ARP traffic.
390
For an active slave, the validation checks ARP replies to confirm
391
that they were generated by an arp_ip_target. Since backup slaves
392
do not typically receive these replies, the validation performed
393
for backup slaves is on the broadcast ARP request sent out via the
394
active slave. It is possible that some switch or network
395
configurations may result in situations wherein the backup slaves
396
do not receive the ARP requests; in such a situation, validation
397
of backup slaves must be disabled.
399
The validation of ARP requests on backup slaves is mainly helping
400
bonding to decide which slaves are more likely to work in case of
401
the active slave failure, it doesn't really guarantee that the
402
backup slave will work if it's selected as the next active slave.
404
Validation is useful in network configurations in which multiple
405
bonding hosts are concurrently issuing ARPs to one or more targets
406
beyond a common switch. Should the link between the switch and
407
target fail (but not the switch itself), the probe traffic
408
generated by the multiple bonding instances will fool the standard
409
ARP monitor into considering the links as still up. Use of
410
validation can resolve this, as the ARP monitor will only consider
411
ARP requests and replies associated with its own instance of
412
bonding.
414
Filtering:
416
Enabling filtering causes the ARP monitor to only use incoming ARP
417
packets for link availability purposes. Arriving packets that are
418
not ARPs are delivered normally, but do not count when determining
419
if a slave is available.
421
Filtering operates by only considering the reception of ARP
422
packets (any ARP packet, regardless of source or destination) when
423
determining if a slave has received traffic for link availability
424
purposes.
426
Filtering is useful in network configurations in which significant
427
levels of third party broadcast traffic would fool the standard
428
ARP monitor into considering the links as still up. Use of
429
filtering can resolve this, as only ARP traffic is considered for
430
link availability purposes.
432
This option was added in bonding version 3.1.0.
434
arp_all_targets
436
Specifies the quantity of arp_ip_targets that must be reachable
437
in order for the ARP monitor to consider a slave as being up.
438
This option affects only active-backup mode for slaves with
439
arp_validation enabled.
441
Possible values are:
443
any or 0
445
consider the slave up only when any of the arp_ip_targets
446
is reachable
448
all or 1
450
consider the slave up only when all of the arp_ip_targets
451
are reachable
453
arp_missed_max
455
Specifies the number of arp_interval monitor checks that must
456
fail in order for an interface to be marked down by the ARP monitor.
458
In order to provide orderly failover semantics, backup interfaces
459
are permitted an extra monitor check (i.e., they must fail
460
arp_missed_max + 1 times before being marked down).
462
The default value is 2, and the allowable range is 1 - 255.
464
coupled_control
466
Specifies whether the LACP state machine's MUX in the 802.3ad mode
467
should have separate Collecting and Distributing states.
469
This is by implementing the independent control state machine per
470
IEEE 802.1AX-2008 5.4.15 in addition to the existing coupled control
471
state machine.
473
The default value is 1. This setting does not separate the Collecting
474
and Distributing states, maintaining the bond in coupled control.
476
downdelay
478
Specifies the time, in milliseconds, to wait before disabling
479
a slave after a link failure has been detected. This option
480
is only valid for the miimon link monitor. The downdelay
481
value should be a multiple of the miimon value; if not, it
482
will be rounded down to the nearest multiple. The default
483
value is 0.
485
fail_over_mac
487
Specifies whether active-backup mode should set all slaves to
488
the same MAC address at enslavement (the traditional
489
behavior), or, when enabled, perform special handling of the
490
bond's MAC address in accordance with the selected policy.
492
Possible values are:
494
none or 0
496
This setting disables fail_over_mac, and causes
497
bonding to set all slaves of an active-backup bond to
498
the same MAC address at enslavement time. This is the
499
default.
501
active or 1
503
The "active" fail_over_mac policy indicates that the
504
MAC address of the bond should always be the MAC
505
address of the currently active slave. The MAC
506
address of the slaves is not changed; instead, the MAC
507
address of the bond changes during a failover.
509
This policy is useful for devices that cannot ever
510
alter their MAC address, or for devices that refuse
511
incoming broadcasts with their own source MAC (which
512
interferes with the ARP monitor).
514
The down side of this policy is that every device on
515
the network must be updated via gratuitous ARP,
516
vs. just updating a switch or set of switches (which
517
often takes place for any traffic, not just ARP
518
traffic, if the switch snoops incoming traffic to
519
update its tables) for the traditional method. If the
520
gratuitous ARP is lost, communication may be
521
disrupted.
523
When this policy is used in conjunction with the mii
524
monitor, devices which assert link up prior to being
525
able to actually transmit and receive are particularly
526
susceptible to loss of the gratuitous ARP, and an
527
appropriate updelay setting may be required.
529
follow or 2
531
The "follow" fail_over_mac policy causes the MAC
532
address of the bond to be selected normally (normally
533
the MAC address of the first slave added to the bond).
534
However, the second and subsequent slaves are not set
535
to this MAC address while they are in a backup role; a
536
slave is programmed with the bond's MAC address at
537
failover time (and the formerly active slave receives
538
the newly active slave's MAC address).
540
This policy is useful for multiport devices that
541
either become confused or incur a performance penalty
542
when multiple ports are programmed with the same MAC
543
address.
546
The default policy is none, unless the first slave cannot
547
change its MAC address, in which case the active policy is
548
selected by default.
550
This option may be modified via sysfs only when no slaves are
551
present in the bond.
553
This option was added in bonding version 3.2.0. The "follow"
554
policy was added in bonding version 3.3.0.
556
lacp_active
557
Option specifying whether to send LACPDU frames periodically.
559
off or 0
560
LACPDU frames acts as "speak when spoken to".
562
on or 1
563
LACPDU frames are sent along the configured links
564
periodically. See lacp_rate for more details.
566
The default is on.
568
lacp_rate
570
Option specifying the rate in which we'll ask our link partner
571
to transmit LACPDU packets in 802.3ad mode. Possible values
572
are:
574
slow or 0
575
Request partner to transmit LACPDUs every 30 seconds
577
fast or 1
578
Request partner to transmit LACPDUs every 1 second
580
The default is slow.
582
broadcast_neighbor
584
Option specifying whether to broadcast ARP/ND packets to all
585
active slaves. This option has no effect in modes other than
586
802.3ad mode. The default is off (0).
588
max_bonds
590
Specifies the number of bonding devices to create for this
591
instance of the bonding driver. E.g., if max_bonds is 3, and
592
the bonding driver is not already loaded, then bond0, bond1
593
and bond2 will be created. The default value is 1. Specifying
594
a value of 0 will load bonding, but will not create any devices.
596
miimon
598
Specifies the MII link monitoring frequency in milliseconds.
599
This determines how often the link state of each slave is
600
inspected for link failures. A value of zero disables MII
601
link monitoring. A value of 100 is a good starting point.
603
The default value is 100 if arp_interval is not set.
605
min_links
607
Specifies the minimum number of links that must be active before
608
asserting carrier. It is similar to the Cisco EtherChannel min-links
609
feature. This allows setting the minimum number of member ports that
610
must be up (link-up state) before marking the bond device as up
611
(carrier on). This is useful for situations where higher level services
612
such as clustering want to ensure a minimum number of low bandwidth
613
links are active before switchover. This option only affect 802.3ad
614
mode.
616
The default value is 0. This will cause carrier to be asserted (for
617
802.3ad mode) whenever there is an active aggregator, regardless of the
618
number of available links in that aggregator. Note that, because an
619
aggregator cannot be active without at least one available link,
620
setting this option to 0 or to 1 has the exact same effect.
622
mode
624
Specifies one of the bonding policies. The default is
625
balance-rr (round robin). Possible values are:
627
balance-rr or 0
629
Round-robin policy: Transmit packets in sequential
630
order from the first available slave through the
631
last. This mode provides load balancing and fault
632
tolerance.
634
active-backup or 1
636
Active-backup policy: Only one slave in the bond is
637
active. A different slave becomes active if, and only
638
if, the active slave fails. The bond's MAC address is
639
externally visible on only one port (network adapter)
640
to avoid confusing the switch.
642
In bonding version 2.6.2 or later, when a failover
643
occurs in active-backup mode, bonding will issue one
644
or more gratuitous ARPs on the newly active slave.
645
One gratuitous ARP is issued for the bonding master
646
interface and each VLAN interfaces configured above
647
it, provided that the interface has at least one IP
648
address configured. Gratuitous ARPs issued for VLAN
649
interfaces are tagged with the appropriate VLAN id.
651
This mode provides fault tolerance. The primary
652
option, documented below, affects the behavior of this
653
mode.
655
balance-xor or 2
657
XOR policy: Transmit based on the selected transmit
658
hash policy. The default policy is a simple [(source
659
MAC address XOR'd with destination MAC address XOR
660
packet type ID) modulo slave count]. Alternate transmit
661
policies may be selected via the xmit_hash_policy option,
662
described below.
664
This mode provides load balancing and fault tolerance.
666
broadcast or 3
668
Broadcast policy: transmits everything on all slave
669
interfaces. This mode provides fault tolerance.
671
802.3ad or 4
673
IEEE 802.3ad Dynamic link aggregation. Creates
674
aggregation groups that share the same speed and
675
duplex settings. Utilizes all slaves in the active
676
aggregator according to the 802.3ad specification.
678
Slave selection for outgoing traffic is done according
679
to the transmit hash policy, which may be changed from
680
the default simple XOR policy via the xmit_hash_policy
681
option, documented below. Note that not all transmit
682
policies may be 802.3ad compliant, particularly in
683
regards to the packet mis-ordering requirements of
684
section 43.2.4 of the 802.3ad standard. Differing
685
peer implementations will have varying tolerances for
686
noncompliance.
688
Prerequisites:
690
1. Ethtool support in the base drivers for retrieving
691
the speed and duplex of each slave.
693
2. A switch that supports IEEE 802.3ad Dynamic link
694
aggregation.
696
Most switches will require some type of configuration
697
to enable 802.3ad mode.
699
balance-tlb or 5
701
Adaptive transmit load balancing: channel bonding that
702
does not require any special switch support.
704
In tlb_dynamic_lb=1 mode; the outgoing traffic is
705
distributed according to the current load (computed
706
relative to the speed) on each slave.
708
In tlb_dynamic_lb=0 mode; the load balancing based on
709
current load is disabled and the load is distributed
710
only using the hash distribution.
712
Incoming traffic is received by the current slave.
713
If the receiving slave fails, another slave takes over
714
the MAC address of the failed receiving slave.
716
Prerequisite:
718
Ethtool support in the base drivers for retrieving the
719
speed of each slave.
721
balance-alb or 6
723
Adaptive load balancing: includes balance-tlb plus
724
receive load balancing (rlb) for IPV4 traffic, and
725
does not require any special switch support. The
726
receive load balancing is achieved by ARP negotiation.
727
The bonding driver intercepts the ARP Replies sent by
728
the local system on their way out and overwrites the
729
source hardware address with the unique hardware
730
address of one of the slaves in the bond such that
731
different peers use different hardware addresses for
732
the server.
734
Receive traffic from connections created by the server
735
is also balanced. When the local system sends an ARP
736
Request the bonding driver copies and saves the peer's
737
IP information from the ARP packet. When the ARP
738
Reply arrives from the peer, its hardware address is
739
retrieved and the bonding driver initiates an ARP
740
reply to this peer assigning it to one of the slaves
741
in the bond. A problematic outcome of using ARP
742
negotiation for balancing is that each time that an
743
ARP request is broadcast it uses the hardware address
744
of the bond. Hence, peers learn the hardware address
745
of the bond and the balancing of receive traffic
746
collapses to the current slave. This is handled by
747
sending updates (ARP Replies) to all the peers with
748
their individually assigned hardware address such that
749
the traffic is redistributed. Receive traffic is also
750
redistributed when a new slave is added to the bond
751
and when an inactive slave is re-activated. The
752
receive load is distributed sequentially (round robin)
753
among the group of highest speed slaves in the bond.
755
When a link is reconnected or a new slave joins the
756
bond the receive traffic is redistributed among all
757
active slaves in the bond by initiating ARP Replies
758
with the selected MAC address to each of the
759
clients. The updelay parameter (detailed below) must
760
be set to a value equal or greater than the switch's
761
forwarding delay so that the ARP Replies sent to the
762
peers will not be blocked by the switch.
764
Prerequisites:
766
1. Ethtool support in the base drivers for retrieving
767
the speed of each slave.
769
2. Base driver support for setting the hardware
770
address of a device while it is open. This is
771
required so that there will always be one slave in the
772
team using the bond hardware address (the
773
curr_active_slave) while having a unique hardware
774
address for each slave in the bond. If the
775
curr_active_slave fails its hardware address is
776
swapped with the new curr_active_slave that was
777
chosen.
779
num_grat_arp,
780
num_unsol_na
782
Specify the number of peer notifications (gratuitous ARPs and
783
unsolicited IPv6 Neighbor Advertisements) to be issued after a
784
failover event. As soon as the link is up on the new slave
785
(possibly immediately) a peer notification is sent on the
786
bonding device and each VLAN sub-device. This is repeated at
787
the rate specified by peer_notif_delay if the number is
788
greater than 1.
790
The valid range is 0 - 255; the default value is 1. These options
791
affect the active-backup or 802.3ad (broadcast_neighbor enabled) mode.
792
These options were added for bonding versions 3.3.0 and 3.4.0
793
respectively.
795
From Linux 3.0 and bonding version 3.7.1, these notifications
796
are generated by the ipv4 and ipv6 code and the numbers of
797
repetitions cannot be set independently.
799
packets_per_slave
801
Specify the number of packets to transmit through a slave before
802
moving to the next one. When set to 0 then a slave is chosen at
803
random.
805
The valid range is 0 - 65535; the default value is 1. This option
806
has effect only in balance-rr mode.
808
peer_notif_delay
810
Specify the delay, in milliseconds, between each peer
811
notification (gratuitous ARP and unsolicited IPv6 Neighbor
812
Advertisement) when they are issued after a failover event.
813
This delay should be a multiple of the MII link monitor interval
814
(miimon).
816
The valid range is 0 - 300000. The default value is 0, which means
817
to match the value of the MII link monitor interval.
819
prio
820
Slave priority. A higher number means higher priority.
821
The primary slave has the highest priority. This option also
822
follows the primary_reselect rules.
824
This option could only be configured via netlink, and is only valid
825
for active-backup(1), balance-tlb (5) and balance-alb (6) mode.
826
The valid value range is a signed 32 bit integer.
828
The default value is 0.
830
primary
832
A string (eth0, eth2, etc) specifying which slave is the
833
primary device. The specified device will always be the
834
active slave while it is available. Only when the primary is
835
off-line will alternate devices be used. This is useful when
836
one slave is preferred over another, e.g., when one slave has
837
higher throughput than another.
839
The primary option is only valid for active-backup(1),
840
balance-tlb (5) and balance-alb (6) mode.
842
primary_reselect
844
Specifies the reselection policy for the primary slave. This
845
affects how the primary slave is chosen to become the active slave
846
when failure of the active slave or recovery of the primary slave
847
occurs. This option is designed to prevent flip-flopping between
848
the primary slave and other slaves. Possible values are:
850
always or 0 (default)
852
The primary slave becomes the active slave whenever it
853
comes back up.
855
better or 1
857
The primary slave becomes the active slave when it comes
858
back up, if the speed and duplex of the primary slave is
859
better than the speed and duplex of the current active
860
slave.
862
failure or 2
864
The primary slave becomes the active slave only if the
865
current active slave fails and the primary slave is up.
867
The primary_reselect setting is ignored in two cases:
869
If no slaves are active, the first slave to recover is
870
made the active slave.
872
When initially enslaved, the primary slave is always made
873
the active slave.
875
Changing the primary_reselect policy via sysfs will cause an
876
immediate selection of the best active slave according to the new
877
policy. This may or may not result in a change of the active
878
slave, depending upon the circumstances.
880
This option was added for bonding version 3.6.0.
882
tlb_dynamic_lb
884
Specifies if dynamic shuffling of flows is enabled in tlb
885
or alb mode. The value has no effect on any other modes.
887
The default behavior of tlb mode is to shuffle active flows across
888
slaves based on the load in that interval. This gives nice lb
889
characteristics but can cause packet reordering. If re-ordering is
890
a concern use this variable to disable flow shuffling and rely on
891
load balancing provided solely by the hash distribution.
892
xmit-hash-policy can be used to select the appropriate hashing for
893
the setup.
895
The sysfs entry can be used to change the setting per bond device
896
and the initial value is derived from the module parameter. The
897
sysfs entry is allowed to be changed only if the bond device is
898
down.
900
The default value is "1" that enables flow shuffling while value "0"
901
disables it. This option was added in bonding driver 3.7.1
904
updelay
906
Specifies the time, in milliseconds, to wait before enabling a
907
slave after a link recovery has been detected. This option is
908
only valid for the miimon link monitor. The updelay value
909
should be a multiple of the miimon value; if not, it will be
910
rounded down to the nearest multiple. The default value is 0.
912
use_carrier
914
Obsolete option that previously selected between MII /
915
ETHTOOL ioctls and netif_carrier_ok() to determine link
916
state.
918
All link state checks are now done with netif_carrier_ok().
920
For backwards compatibility, this option's value may be inspected
921
or set. The only valid setting is 1.
923
xmit_hash_policy
925
Selects the transmit hash policy to use for slave selection in
926
balance-xor, 802.3ad, and tlb modes. Possible values are:
928
layer2
930
Uses XOR of hardware MAC addresses and packet type ID
931
field to generate the hash. The formula is
933
hash = source MAC[5] XOR destination MAC[5] XOR packet type ID
934
slave number = hash modulo slave count
936
This algorithm will place all traffic to a particular
937
network peer on the same slave.
939
This algorithm is 802.3ad compliant.
941
layer2+3
943
This policy uses a combination of layer2 and layer3
944
protocol information to generate the hash.
946
Uses XOR of hardware MAC addresses and IP addresses to
947
generate the hash. The formula is
949
hash = source MAC[5] XOR destination MAC[5] XOR packet type ID
950
hash = hash XOR source IP XOR destination IP
951
hash = hash XOR (hash RSHIFT 16)
952
hash = hash XOR (hash RSHIFT 8)
953
And then hash is reduced modulo slave count.
955
If the protocol is IPv6 then the source and destination
956
addresses are first hashed using ipv6_addr_hash.
958
This algorithm will place all traffic to a particular
959
network peer on the same slave. For non-IP traffic,
960
the formula is the same as for the layer2 transmit
961
hash policy.
963
This policy is intended to provide a more balanced
964
distribution of traffic than layer2 alone, especially
965
in environments where a layer3 gateway device is
966
required to reach most destinations.
968
This algorithm is 802.3ad compliant.
970
layer3+4
972
This policy uses upper layer protocol information,
973
when available, to generate the hash. This allows for
974
traffic to a particular network peer to span multiple
975
slaves, although a single connection will not span
976
multiple slaves.
978
The formula for unfragmented TCP and UDP packets is
980
hash = source port, destination port (as in the header)
981
hash = hash XOR source IP XOR destination IP
982
hash = hash XOR (hash RSHIFT 16)
983
hash = hash XOR (hash RSHIFT 8)
984
hash = hash RSHIFT 1
985
And then hash is reduced modulo slave count.
987
If the protocol is IPv6 then the source and destination
988
addresses are first hashed using ipv6_addr_hash.
990
For fragmented TCP or UDP packets and all other IPv4 and
991
IPv6 protocol traffic, the source and destination port
992
information is omitted. For non-IP traffic, the
993
formula is the same as for the layer2 transmit hash
994
policy.
996
This algorithm is not fully 802.3ad compliant. A
997
single TCP or UDP conversation containing both
998
fragmented and unfragmented packets will see packets
999
striped across two interfaces. This may result in out
1000
of order delivery. Most traffic types will not meet
1001
this criteria, as TCP rarely fragments traffic, and
1002
most UDP traffic is not involved in extended
1003
conversations. Other implementations of 802.3ad may
1004
or may not tolerate this noncompliance.
1006
encap2+3
1008
This policy uses the same formula as layer2+3 but it
1009
relies on skb_flow_dissect to obtain the header fields
1010
which might result in the use of inner headers if an
1011
encapsulation protocol is used. For example this will
1012
improve the performance for tunnel users because the
1013
packets will be distributed according to the encapsulated
1014
flows.
1016
encap3+4
1018
This policy uses the same formula as layer3+4 but it
1019
relies on skb_flow_dissect to obtain the header fields
1020
which might result in the use of inner headers if an
1021
encapsulation protocol is used. For example this will
1022
improve the performance for tunnel users because the
1023
packets will be distributed according to the encapsulated
1024
flows.
1026
vlan+srcmac
1028
This policy uses a very rudimentary vlan ID and source mac
1029
hash to load-balance traffic per-vlan, with failover
1030
should one leg fail. The intended use case is for a bond
1031
shared by multiple virtual machines, all configured to
1032
use their own vlan, to give lacp-like functionality
1033
without requiring lacp-capable switching hardware.
1035
The formula for the hash is simply
1037
hash = (vlan ID) XOR (source MAC vendor) XOR (source MAC dev)
1039
The default value is layer2. This option was added in bonding
1040
version 2.6.3. In earlier versions of bonding, this parameter
1041
does not exist, and the layer2 policy is the only policy. The
1042
layer2+3 value was added for bonding version 3.2.2.
1044
resend_igmp
1046
Specifies the number of IGMP membership reports to be issued after
1047
a failover event. One membership report is issued immediately after
1048
the failover, subsequent packets are sent in each 200ms interval.
1050
The valid range is 0 - 255; the default value is 1. A value of 0
1051
prevents the IGMP membership report from being issued in response
1052
to the failover event.
1054
This option is useful for bonding modes balance-rr (0), active-backup
1055
(1), balance-tlb (5) and balance-alb (6), in which a failover can
1056
switch the IGMP traffic from one slave to another. Therefore a fresh
1057
IGMP report must be issued to cause the switch to forward the incoming
1058
IGMP traffic over the newly selected slave.
1060
This option was added for bonding version 3.7.0.
1062
lp_interval
1064
Specifies the number of seconds between instances where the bonding
1065
driver sends learning packets to each slaves peer switch.
1067
The valid range is 1 - 0x7fffffff; the default value is 1. This Option
1068
has effect only in balance-tlb and balance-alb modes.
1070
3. Configuring Bonding Devices
1071
==============================
1073
You can configure bonding using either your distro's network
1074
initialization scripts, or manually using either iproute2 or the
1075
sysfs interface. Distros generally use one of three packages for the
1076
network initialization scripts: initscripts, sysconfig or interfaces.
1077
Recent versions of these packages have support for bonding, while older
1078
versions do not.
1080
We will first describe the options for configuring bonding for
1081
distros using versions of initscripts, sysconfig and interfaces with full
1082
or partial support for bonding, then provide information on enabling
1083
bonding without support from the network initialization scripts (i.e.,
1084
older versions of initscripts or sysconfig).
1086
If you're unsure whether your distro uses sysconfig,
1087
initscripts or interfaces, or don't know if it's new enough, have no fear.
1088
Determining this is fairly straightforward.
1090
First, look for a file called interfaces in /etc/network directory.
1091
If this file is present in your system, then your system use interfaces. See
1092
Configuration with Interfaces Support.
1094
Else, issue the command::
1096
$ rpm -qf /sbin/ifup
1098
It will respond with a line of text starting with either
1099
"initscripts" or "sysconfig," followed by some numbers. This is the
1100
package that provides your network initialization scripts.
1102
Next, to determine if your installation supports bonding,
1103
issue the command::
1105
$ grep ifenslave /sbin/ifup
1107
If this returns any matches, then your initscripts or
1108
sysconfig has support for bonding.
1110
3.1 Configuration with Sysconfig Support
1111
----------------------------------------
1113
This section applies to distros using a version of sysconfig
1114
with bonding support, for example, SuSE Linux Enterprise Server 9.
1116
SuSE SLES 9's networking configuration system does support
1117
bonding, however, at this writing, the YaST system configuration
1118
front end does not provide any means to work with bonding devices.
1119
Bonding devices can be managed by hand, however, as follows.
1121
First, if they have not already been configured, configure the
1122
slave devices. On SLES 9, this is most easily done by running the
1123
yast2 sysconfig configuration utility. The goal is for to create an
1124
ifcfg-id file for each slave device. The simplest way to accomplish
1125
this is to configure the devices for DHCP (this is only to get the
1126
file ifcfg-id file created; see below for some issues with DHCP). The
1127
name of the configuration file for each device will be of the form::
1129
ifcfg-id-xx:xx:xx:xx:xx:xx
1131
Where the "xx" portion will be replaced with the digits from
1132
the device's permanent MAC address.
1134
Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
1135
created, it is necessary to edit the configuration files for the slave
1136
devices (the MAC addresses correspond to those of the slave devices).
1137
Before editing, the file will contain multiple lines, and will look
1138
something like this::
1140
BOOTPROTO='dhcp'
1141
STARTMODE='on'
1142
USERCTL='no'
1143
UNIQUE='XNzu.WeZGOGF+4wE'
1144
_nm_name='bus-pci-0001:61:01.0'
1146
Change the BOOTPROTO and STARTMODE lines to the following::
1148
BOOTPROTO='none'
1149
STARTMODE='off'
1151
Do not alter the UNIQUE or _nm_name lines. Remove any other
1152
lines (USERCTL, etc).
1154
Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified,
1155
it's time to create the configuration file for the bonding device
1156
itself. This file is named ifcfg-bondX, where X is the number of the
1157
bonding device to create, starting at 0. The first such file is
1158
ifcfg-bond0, the second is ifcfg-bond1, and so on. The sysconfig
1159
network configuration system will correctly start multiple instances
1160
of bonding.
1162
The contents of the ifcfg-bondX file is as follows::
1164
BOOTPROTO="static"
1165
BROADCAST="10.0.2.255"
1166
IPADDR="10.0.2.10"
1167
NETMASK="255.255.0.0"
1168
NETWORK="10.0.2.0"
1169
REMOTE_IPADDR=""
1170
STARTMODE="onboot"
1171
BONDING_MASTER="yes"
1172
BONDING_MODULE_OPTS="mode=active-backup miimon=100"
1173
BONDING_SLAVE0="eth0"
1174
BONDING_SLAVE1="bus-pci-0000:06:08.1"
1176
Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
1177
values with the appropriate values for your network.
1179
The STARTMODE specifies when the device is brought online.
1180
The possible values are:
1182
======== ======================================================
1183
onboot The device is started at boot time. If you're not
1184
sure, this is probably what you want.
1186
manual The device is started only when ifup is called
1187
manually. Bonding devices may be configured this
1188
way if you do not wish them to start automatically
1189
at boot for some reason.
1191
hotplug The device is started by a hotplug event. This is not
1192
a valid choice for a bonding device.
1194
off or The device configuration is ignored.
1195
ignore
1196
======== ======================================================
1198
The line BONDING_MASTER='yes' indicates that the device is a
1199
bonding master device. The only useful value is "yes."
1201
The contents of BONDING_MODULE_OPTS are supplied to the
1202
instance of the bonding module for this device. Specify the options
1203
for the bonding mode, link monitoring, and so on here. Do not include
1204
the max_bonds bonding parameter; this will confuse the configuration
1205
system if you have multiple bonding devices.
1207
Finally, supply one BONDING_SLAVEn="slave device" for each
1208
slave. where "n" is an increasing value, one for each slave. The
1209
"slave device" is either an interface name, e.g., "eth0", or a device
1210
specifier for the network device. The interface name is easier to
1211
find, but the ethN names are subject to change at boot time if, e.g.,
1212
a device early in the sequence has failed. The device specifiers
1213
(bus-pci-0000:06:08.1 in the example above) specify the physical
1214
network device, and will not change unless the device's bus location
1215
changes (for example, it is moved from one PCI slot to another). The
1216
example above uses one of each type for demonstration purposes; most
1217
configurations will choose one or the other for all slave devices.
1219
When all configuration files have been modified or created,
1220
networking must be restarted for the configuration changes to take
1221
effect. This can be accomplished via the following::
1223
# /etc/init.d/network restart
1225
Note that the network control script (/sbin/ifdown) will
1226
remove the bonding module as part of the network shutdown processing,
1227
so it is not necessary to remove the module by hand if, e.g., the
1228
module parameters have changed.
1230
Also, at this writing, YaST/YaST2 will not manage bonding
1231
devices (they do not show bonding interfaces on its list of network
1232
devices). It is necessary to edit the configuration file by hand to
1233
change the bonding configuration.
1235
Additional general options and details of the ifcfg file
1236
format can be found in an example ifcfg template file::
1238
/etc/sysconfig/network/ifcfg.template
1240
Note that the template does not document the various ``BONDING_*``
1241
settings described above, but does describe many of the other options.
1243
3.1.1 Using DHCP with Sysconfig
1244
-------------------------------
1246
Under sysconfig, configuring a device with BOOTPROTO='dhcp'
1247
will cause it to query DHCP for its IP address information. At this
1248
writing, this does not function for bonding devices; the scripts
1249
attempt to obtain the device address from DHCP prior to adding any of
1250
the slave devices. Without active slaves, the DHCP requests are not
1251
sent to the network.
1253
3.1.2 Configuring Multiple Bonds with Sysconfig
1254
-----------------------------------------------
1256
The sysconfig network initialization system is capable of
1257
handling multiple bonding devices. All that is necessary is for each
1258
bonding instance to have an appropriately configured ifcfg-bondX file
1259
(as described above). Do not specify the "max_bonds" parameter to any
1260
instance of bonding, as this will confuse sysconfig. If you require
1261
multiple bonding devices with identical parameters, create multiple
1262
ifcfg-bondX files.
1264
Because the sysconfig scripts supply the bonding module
1265
options in the ifcfg-bondX file, it is not necessary to add them to
1266
the system ``/etc/modules.d/*.conf`` configuration files.
1268
3.2 Configuration with Initscripts Support
1269
------------------------------------------
1271
This section applies to distros using a recent version of
1272
initscripts with bonding support, for example, Red Hat Enterprise Linux
1273
version 3 or later, Fedora, etc. On these systems, the network
1274
initialization scripts have knowledge of bonding, and can be configured to
1275
control bonding devices. Note that older versions of the initscripts
1276
package have lower levels of support for bonding; this will be noted where
1277
applicable.
1279
These distros will not automatically load the network adapter
1280
driver unless the ethX device is configured with an IP address.
1281
Because of this constraint, users must manually configure a
1282
network-script file for all physical adapters that will be members of
1283
a bondX link. Network script files are located in the directory:
1285
/etc/sysconfig/network-scripts
1287
The file name must be prefixed with "ifcfg-eth" and suffixed
1288
with the adapter's physical adapter number. For example, the script
1289
for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0.
1290
Place the following text in the file::
1292
DEVICE=eth0
1293
USERCTL=no
1294
ONBOOT=yes
1295
MASTER=bond0
1296
SLAVE=yes
1297
BOOTPROTO=none
1299
The DEVICE= line will be different for every ethX device and
1300
must correspond with the name of the file, i.e., ifcfg-eth1 must have
1301
a device line of DEVICE=eth1. The setting of the MASTER= line will
1302
also depend on the final bonding interface name chosen for your bond.
1303
As with other network devices, these typically start at 0, and go up
1304
one for each device, i.e., the first bonding instance is bond0, the
1305
second is bond1, and so on.
1307
Next, create a bond network script. The file name for this
1308
script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is
1309
the number of the bond. For bond0 the file is named "ifcfg-bond0",
1310
for bond1 it is named "ifcfg-bond1", and so on. Within that file,
1311
place the following text::
1313
DEVICE=bond0
1314
IPADDR=192.168.1.1
1315
NETMASK=255.255.255.0
1316
NETWORK=192.168.1.0
1317
BROADCAST=192.168.1.255
1318
ONBOOT=yes
1319
BOOTPROTO=none
1320
USERCTL=no
1322
Be sure to change the networking specific lines (IPADDR,
1323
NETMASK, NETWORK and BROADCAST) to match your network configuration.
1325
For later versions of initscripts, such as that found with Fedora
1326
7 (or later) and Red Hat Enterprise Linux version 5 (or later), it is possible,
1327
and, indeed, preferable, to specify the bonding options in the ifcfg-bond0
1328
file, e.g. a line of the format::
1330
BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=192.168.1.254"
1332
will configure the bond with the specified options. The options
1333
specified in BONDING_OPTS are identical to the bonding module parameters
1334
except for the arp_ip_target field when using versions of initscripts older
1335
than and 8.57 (Fedora 8) and 8.45.19 (Red Hat Enterprise Linux 5.2). When
1336
using older versions each target should be included as a separate option and
1337
should be preceded by a '+' to indicate it should be added to the list of
1338
queried targets, e.g.,::
1340
arp_ip_target=+192.168.1.1 arp_ip_target=+192.168.1.2
1342
is the proper syntax to specify multiple targets. When specifying
1343
options via BONDING_OPTS, it is not necessary to edit
1344
``/etc/modprobe.d/*.conf``.
1346
For even older versions of initscripts that do not support
1347
BONDING_OPTS, it is necessary to edit /etc/modprobe.d/*.conf, depending upon
1348
your distro) to load the bonding module with your desired options when the
1349
bond0 interface is brought up. The following lines in /etc/modprobe.d/*.conf
1350
will load the bonding module, and select its options:
1352
alias bond0 bonding
1353
options bond0 mode=balance-alb miimon=100
1355
Replace the sample parameters with the appropriate set of
1356
options for your configuration.
1358
Finally run "/etc/rc.d/init.d/network restart" as root. This
1359
will restart the networking subsystem and your bond link should be now
1360
up and running.
1362
3.2.1 Using DHCP with Initscripts
1363
---------------------------------
1365
Recent versions of initscripts (the versions supplied with Fedora
1366
Core 3 and Red Hat Enterprise Linux 4, or later versions, are reported to
1367
work) have support for assigning IP information to bonding devices via
1368
DHCP.
1370
To configure bonding for DHCP, configure it as described
1371
above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp"
1372
and add a line consisting of "TYPE=Bonding". Note that the TYPE value
1373
is case sensitive.
1375
3.2.2 Configuring Multiple Bonds with Initscripts
1376
-------------------------------------------------
1378
Initscripts packages that are included with Fedora 7 and Red Hat
1379
Enterprise Linux 5 support multiple bonding interfaces by simply
1380
specifying the appropriate BONDING_OPTS= in ifcfg-bondX where X is the
1381
number of the bond. This support requires sysfs support in the kernel,
1382
and a bonding driver of version 3.0.0 or later. Other configurations may
1383
not support this method for specifying multiple bonding interfaces; for
1384
those instances, see the "Configuring Multiple Bonds Manually" section,
1385
below.
1387
3.3 Configuring Bonding Manually with iproute2
1388
-----------------------------------------------
1390
This section applies to distros whose network initialization
1391
scripts (the sysconfig or initscripts package) do not have specific
1392
knowledge of bonding. One such distro is SuSE Linux Enterprise Server
1393
version 8.
1395
The general method for these systems is to place the bonding
1396
module parameters into a config file in /etc/modprobe.d/ (as
1397
appropriate for the installed distro), then add modprobe and/or
1398
`ip link` commands to the system's global init script. The name of
1399
the global init script differs; for sysconfig, it is
1400
/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
1402
For example, if you wanted to make a simple bond of two e100
1403
devices (presumed to be eth0 and eth1), and have it persist across
1404
reboots, edit the appropriate file (/etc/init.d/boot.local or
1405
/etc/rc.d/rc.local), and add the following::
1407
modprobe bonding mode=balance-alb miimon=100
1408
modprobe e100
1409
ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
1410
ip link set eth0 master bond0
1411
ip link set eth1 master bond0
1413
Replace the example bonding module parameters and bond0
1414
network configuration (IP address, netmask, etc) with the appropriate
1415
values for your configuration.
1417
Unfortunately, this method will not provide support for the
1418
ifup and ifdown scripts on the bond devices. To reload the bonding
1419
configuration, it is necessary to run the initialization script, e.g.,::
1421
# /etc/init.d/boot.local
1423
or::
1425
# /etc/rc.d/rc.local
1427
It may be desirable in such a case to create a separate script
1428
which only initializes the bonding configuration, then call that
1429
separate script from within boot.local. This allows for bonding to be
1430
enabled without re-running the entire global init script.
1432
To shut down the bonding devices, it is necessary to first
1433
mark the bonding device itself as being down, then remove the
1434
appropriate device driver modules. For our example above, you can do
1435
the following::
1437
# ifconfig bond0 down
1438
# rmmod bonding
1439
# rmmod e100
1441
Again, for convenience, it may be desirable to create a script
1442
with these commands.
1445
3.3.1 Configuring Multiple Bonds Manually
1446
-----------------------------------------
1448
This section contains information on configuring multiple
1449
bonding devices with differing options for those systems whose network
1450
initialization scripts lack support for configuring multiple bonds.
1452
If you require multiple bonding devices, but all with the same
1453
options, you may wish to use the "max_bonds" module parameter,
1454
documented above.
1456
To create multiple bonding devices with differing options, it is
1457
preferable to use bonding parameters exported by sysfs, documented in the
1458
section below.
1460
For versions of bonding without sysfs support, the only means to
1461
provide multiple instances of bonding with differing options is to load
1462
the bonding driver multiple times. Note that current versions of the
1463
sysconfig network initialization scripts handle this automatically; if
1464
your distro uses these scripts, no special action is needed. See the
1465
section Configuring Bonding Devices, above, if you're not sure about your
1466
network initialization scripts.
1468
To load multiple instances of the module, it is necessary to
1469
specify a different name for each instance (the module loading system
1470
requires that every loaded module, even multiple instances of the same
1471
module, have a unique name). This is accomplished by supplying multiple
1472
sets of bonding options in ``/etc/modprobe.d/*.conf``, for example::
1474
alias bond0 bonding
1475
options bond0 -o bond0 mode=balance-rr miimon=100
1477
alias bond1 bonding
1478
options bond1 -o bond1 mode=balance-alb miimon=50
1480
will load the bonding module two times. The first instance is
1481
named "bond0" and creates the bond0 device in balance-rr mode with an
1482
miimon of 100. The second instance is named "bond1" and creates the
1483
bond1 device in balance-alb mode with an miimon of 50.
1485
In some circumstances (typically with older distributions),
1486
the above does not work, and the second bonding instance never sees
1487
its options. In that case, the second options line can be substituted
1488
as follows::
1490
install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \
1491
mode=balance-alb miimon=50
1493
This may be repeated any number of times, specifying a new and
1494
unique name in place of bond1 for each subsequent instance.
1496
It has been observed that some Red Hat supplied kernels are unable
1497
to rename modules at load time (the "-o bond1" part). Attempts to pass
1498
that option to modprobe will produce an "Operation not permitted" error.
1499
This has been reported on some Fedora Core kernels, and has been seen on
1500
RHEL 4 as well. On kernels exhibiting this problem, it will be impossible
1501
to configure multiple bonds with differing parameters (as they are older
1502
kernels, and also lack sysfs support).
1504
3.4 Configuring Bonding Manually via Sysfs
1505
------------------------------------------
1507
Starting with version 3.0.0, Channel Bonding may be configured
1508
via the sysfs interface. This interface allows dynamic configuration
1509
of all bonds in the system without unloading the module. It also
1510
allows for adding and removing bonds at runtime. Ifenslave is no
1511
longer required, though it is still supported.
1513
Use of the sysfs interface allows you to use multiple bonds
1514
with different configurations without having to reload the module.
1515
It also allows you to use multiple, differently configured bonds when
1516
bonding is compiled into the kernel.
1518
You must have the sysfs filesystem mounted to configure
1519
bonding this way. The examples in this document assume that you
1520
are using the standard mount point for sysfs, e.g. /sys. If your
1521
sysfs filesystem is mounted elsewhere, you will need to adjust the
1522
example paths accordingly.
1524
Creating and Destroying Bonds
1525
-----------------------------
1526
To add a new bond foo::
1528
# echo +foo > /sys/class/net/bonding_masters
1530
To remove an existing bond bar::
1532
# echo -bar > /sys/class/net/bonding_masters
1534
To show all existing bonds::
1536
# cat /sys/class/net/bonding_masters
1538
.. note::
1540
due to 4K size limitation of sysfs files, this list may be
1541
truncated if you have more than a few hundred bonds. This is unlikely
1542
to occur under normal operating conditions.
1544
Adding and Removing Slaves
1545
--------------------------
1546
Interfaces may be enslaved to a bond using the file
1547
/sys/class/net/<bond>/bonding/slaves. The semantics for this file
1548
are the same as for the bonding_masters file.
1550
To enslave interface eth0 to bond bond0::
1552
# ifconfig bond0 up
1553
# echo +eth0 > /sys/class/net/bond0/bonding/slaves
1555
To free slave eth0 from bond bond0::
1557
# echo -eth0 > /sys/class/net/bond0/bonding/slaves
1559
When an interface is enslaved to a bond, symlinks between the
1560
two are created in the sysfs filesystem. In this case, you would get
1561
/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and
1562
/sys/class/net/eth0/master pointing to /sys/class/net/bond0.
1564
This means that you can tell quickly whether or not an
1565
interface is enslaved by looking for the master symlink. Thus:
1566
# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves
1567
will free eth0 from whatever bond it is enslaved to, regardless of
1568
the name of the bond interface.
1570
Changing a Bond's Configuration
1571
-------------------------------
1572
Each bond may be configured individually by manipulating the
1573
files located in /sys/class/net/<bond name>/bonding
1575
The names of these files correspond directly with the command-
1576
line parameters described elsewhere in this file, and, with the
1577
exception of arp_ip_target, they accept the same values. To see the
1578
current setting, simply cat the appropriate file.
1580
A few examples will be given here; for specific usage
1581
guidelines for each parameter, see the appropriate section in this
1582
document.
1584
To configure bond0 for balance-alb mode::
1586
# ifconfig bond0 down
1587
# echo 6 > /sys/class/net/bond0/bonding/mode
1588
- or -
1589
# echo balance-alb > /sys/class/net/bond0/bonding/mode
1591
.. note::
1593
The bond interface must be down before the mode can be changed.
1595
To enable MII monitoring on bond0 with a 1 second interval::
1597
# echo 1000 > /sys/class/net/bond0/bonding/miimon
1599
.. note::
1601
If ARP monitoring is enabled, it will disabled when MII
1602
monitoring is enabled, and vice-versa.
1604
To add ARP targets::
1606
# echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
1607
# echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target
1609
.. note::
1611
up to 16 target addresses may be specified.
1613
To remove an ARP target::
1615
# echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
1617
To configure the interval between learning packet transmits::
1619
# echo 12 > /sys/class/net/bond0/bonding/lp_interval
1621
.. note::
1623
the lp_interval is the number of seconds between instances where
1624
the bonding driver sends learning packets to each slaves peer switch. The
1625
default interval is 1 second.
1627
Example Configuration
1628
---------------------
1629
We begin with the same example that is shown in section 3.3,
1630
executed with sysfs, and without using ifenslave.
1632
To make a simple bond of two e100 devices (presumed to be eth0
1633
and eth1), and have it persist across reboots, edit the appropriate
1634
file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the
1635
following::
1637
modprobe bonding
1638
modprobe e100
1639
echo balance-alb > /sys/class/net/bond0/bonding/mode
1640
ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
1641
echo 100 > /sys/class/net/bond0/bonding/miimon
1642
echo +eth0 > /sys/class/net/bond0/bonding/slaves
1643
echo +eth1 > /sys/class/net/bond0/bonding/slaves
1645
To add a second bond, with two e1000 interfaces in
1646
active-backup mode, using ARP monitoring, add the following lines to
1647
your init script::
1649
modprobe e1000
1650
echo +bond1 > /sys/class/net/bonding_masters
1651
echo active-backup > /sys/class/net/bond1/bonding/mode
1652
ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up
1653
echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target
1654
echo 2000 > /sys/class/net/bond1/bonding/arp_interval
1655
echo +eth2 > /sys/class/net/bond1/bonding/slaves
1656
echo +eth3 > /sys/class/net/bond1/bonding/slaves
1658
3.5 Configuration with Interfaces Support
1659
-----------------------------------------
1661
This section applies to distros which use /etc/network/interfaces file
1662
to describe network interface configuration, most notably Debian and its
1663
derivatives.
1665
The ifup and ifdown commands on Debian don't support bonding out of
1666
the box. The ifenslave-2.6 package should be installed to provide bonding
1667
support. Once installed, this package will provide ``bond-*`` options
1668
to be used into /etc/network/interfaces.
1670
Note that ifenslave-2.6 package will load the bonding module and use
1671
the ifenslave command when appropriate.
1673
Example Configurations
1674
----------------------
1676
In /etc/network/interfaces, the following stanza will configure bond0, in
1677
active-backup mode, with eth0 and eth1 as slaves::
1679
auto bond0
1680
iface bond0 inet dhcp
1681
bond-slaves eth0 eth1
1682
bond-mode active-backup
1683
bond-miimon 100
1684
bond-primary eth0 eth1
1686
If the above configuration doesn't work, you might have a system using
1687
upstart for system startup. This is most notably true for recent
1688
Ubuntu versions. The following stanza in /etc/network/interfaces will
1689
produce the same result on those systems::
1691
auto bond0
1692
iface bond0 inet dhcp
1693
bond-slaves none
1694
bond-mode active-backup
1695
bond-miimon 100
1697
auto eth0
1698
iface eth0 inet manual
1699
bond-master bond0
1700
bond-primary eth0 eth1
1702
auto eth1
1703
iface eth1 inet manual
1704
bond-master bond0
1705
bond-primary eth0 eth1
1707
For a full list of ``bond-*`` supported options in /etc/network/interfaces and
1708
some more advanced examples tailored to you particular distros, see the files in
1709
/usr/share/doc/ifenslave-2.6.
1711
3.6 Overriding Configuration for Special Cases
1712
----------------------------------------------
1714
When using the bonding driver, the physical port which transmits a frame is
1715
typically selected by the bonding driver, and is not relevant to the user or
1716
system administrator. The output port is simply selected using the policies of
1717
the selected bonding mode. On occasion however, it is helpful to direct certain
1718
classes of traffic to certain physical interfaces on output to implement
1719
slightly more complex policies. For example, to reach a web server over a
1720
bonded interface in which eth0 connects to a private network, while eth1
1721
connects via a public network, it may be desirous to bias the bond to send said
1722
traffic over eth0 first, using eth1 only as a fall back, while all other traffic
1723
can safely be sent over either interface. Such configurations may be achieved
1724
using the traffic control utilities inherent in linux.
1726
By default the bonding driver is multiqueue aware and 16 queues are created
1727
when the driver initializes (see Documentation/networking/multiqueue.rst
1728
for details). If more or less queues are desired the module parameter
1729
tx_queues can be used to change this value. There is no sysfs parameter
1730
available as the allocation is done at module init time.
1732
The output of the file /proc/net/bonding/bondX has changed so the output Queue
1733
ID is now printed for each slave::
1735
Bonding Mode: fault-tolerance (active-backup)
1736
Primary Slave: None
1737
Currently Active Slave: eth0
1738
MII Status: up
1739
MII Polling Interval (ms): 0
1740
Up Delay (ms): 0
1741
Down Delay (ms): 0
1743
Slave Interface: eth0
1744
MII Status: up
1745
Link Failure Count: 0
1746
Permanent HW addr: 00:1a:a0:12:8f:cb
1747
Slave queue ID: 0
1749
Slave Interface: eth1
1750
MII Status: up
1751
Link Failure Count: 0
1752
Permanent HW addr: 00:1a:a0:12:8f:cc
1753
Slave queue ID: 2
1755
The queue_id for a slave can be set using the command::
1757
# echo "eth1:2" > /sys/class/net/bond0/bonding/queue_id
1759
Any interface that needs a queue_id set should set it with multiple calls
1760
like the one above until proper priorities are set for all interfaces. On
1761
distributions that allow configuration via initscripts, multiple 'queue_id'
1762
arguments can be added to BONDING_OPTS to set all needed slave queues.
1764
These queue id's can be used in conjunction with the tc utility to configure
1765
a multiqueue qdisc and filters to bias certain traffic to transmit on certain
1766
slave devices. For instance, say we wanted, in the above configuration to
1767
force all traffic bound to 192.168.1.100 to use eth1 in the bond as its output
1768
device. The following commands would accomplish this::
1770
# tc qdisc add dev bond0 handle 1 root multiq
1772
# tc filter add dev bond0 protocol ip parent 1: prio 1 u32 match ip \
1773
dst 192.168.1.100 action skbedit queue_mapping 2
1775
These commands tell the kernel to attach a multiqueue queue discipline to the
1776
bond0 interface and filter traffic enqueued to it, such that packets with a dst
1777
ip of 192.168.1.100 have their output queue mapping value overwritten to 2.
1778
This value is then passed into the driver, causing the normal output path
1779
selection policy to be overridden, selecting instead qid 2, which maps to eth1.
1781
Note that qid values begin at 1. Qid 0 is reserved to initiate to the driver
1782
that normal output policy selection should take place. One benefit to simply
1783
leaving the qid for a slave to 0 is the multiqueue awareness in the bonding
1784
driver that is now present. This awareness allows tc filters to be placed on
1785
slave devices as well as bond devices and the bonding driver will simply act as
1786
a pass-through for selecting output queues on the slave device rather than
1787
output port selection.
1789
This feature first appeared in bonding driver version 3.7.0 and support for
1790
output slave selection was limited to round-robin and active-backup modes.
1792
3.7 Configuring LACP for 802.3ad mode in a more secure way
1793
----------------------------------------------------------
1795
When using 802.3ad bonding mode, the Actor (host) and Partner (switch)
1796
exchange LACPDUs. These LACPDUs cannot be sniffed, because they are
1797
destined to link local mac addresses (which switches/bridges are not
1798
supposed to forward). However, most of the values are easily predictable
1799
or are simply the machine's MAC address (which is trivially known to all
1800
other hosts in the same L2). This implies that other machines in the L2
1801
domain can spoof LACPDU packets from other hosts to the switch and potentially
1802
cause mayhem by joining (from the point of view of the switch) another
1803
machine's aggregate, thus receiving a portion of that hosts incoming
1804
traffic and / or spoofing traffic from that machine themselves (potentially
1805
even successfully terminating some portion of flows). Though this is not
1806
a likely scenario, one could avoid this possibility by simply configuring
1807
few bonding parameters:
1809
(a) ad_actor_system : You can set a random mac-address that can be used for
1810
these LACPDU exchanges. The value can not be either NULL or Multicast.
1811
Also it's preferable to set the local-admin bit. Following shell code
1812
generates a random mac-address as described above::
1814
# sys_mac_addr=$(printf '%02x:%02x:%02x:%02x:%02x:%02x' \
1815
$(( (RANDOM & 0xFE) | 0x02 )) \
1816
$(( RANDOM & 0xFF )) \
1817
$(( RANDOM & 0xFF )) \
1818
$(( RANDOM & 0xFF )) \
1819
$(( RANDOM & 0xFF )) \
1820
$(( RANDOM & 0xFF )))
1821
# echo $sys_mac_addr > /sys/class/net/bond0/bonding/ad_actor_system
1823
(b) ad_actor_sys_prio : Randomize the system priority. The default value
1824
is 65535, but system can take the value from 1 - 65535. Following shell
1825
code generates random priority and sets it::
1827
# sys_prio=$(( 1 + RANDOM + RANDOM ))
1828
# echo $sys_prio > /sys/class/net/bond0/bonding/ad_actor_sys_prio
1830
(c) ad_user_port_key : Use the user portion of the port-key. The default
1831
keeps this empty. These are the upper 10 bits of the port-key and value
1832
ranges from 0 - 1023. Following shell code generates these 10 bits and
1833
sets it::
1835
# usr_port_key=$(( RANDOM & 0x3FF ))
1836
# echo $usr_port_key > /sys/class/net/bond0/bonding/ad_user_port_key
1839
4 Querying Bonding Configuration
1840
=================================
1842
4.1 Bonding Configuration
1843
-------------------------
1845
Each bonding device has a read-only file residing in the
1846
/proc/net/bonding directory. The file contents include information
1847
about the bonding configuration, options and state of each slave.
1849
For example, the contents of /proc/net/bonding/bond0 after the
1850
driver is loaded with parameters of mode=0 and miimon=1000 is
1851
generally as follows::
1853
Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004)
1854
Bonding Mode: load balancing (round-robin)
1855
Currently Active Slave: eth0
1856
MII Status: up
1857
MII Polling Interval (ms): 1000
1858
Up Delay (ms): 0
1859
Down Delay (ms): 0
1861
Slave Interface: eth1
1862
MII Status: up
1863
Link Failure Count: 1
1865
Slave Interface: eth0
1866
MII Status: up
1867
Link Failure Count: 1
1869
The precise format and contents will change depending upon the
1870
bonding configuration, state, and version of the bonding driver.
1872
4.2 Network configuration
1873
-------------------------
1875
The network configuration can be inspected using the ifconfig
1876
command. Bonding devices will have the MASTER flag set; Bonding slave
1877
devices will have the SLAVE flag set. The ifconfig output does not
1878
contain information on which slaves are associated with which masters.
1880
In the example below, the bond0 interface is the master
1881
(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of
1882
bond0 have the same MAC address (HWaddr) as bond0 for all modes except
1883
TLB and ALB that require a unique MAC address for each slave::
1885
# /sbin/ifconfig
1886
bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
1887
inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
1888
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
1889
RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
1890
TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
1891
collisions:0 txqueuelen:0
1893
eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
1894
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
1895
RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
1896
TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
1897
collisions:0 txqueuelen:100
1898
Interrupt:10 Base address:0x1080
1900
eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
1901
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
1902
RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
1903
TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
1904
collisions:0 txqueuelen:100
1905
Interrupt:9 Base address:0x1400
1907
5. Switch Configuration
1908
=======================
1910
For this section, "switch" refers to whatever system the
1911
bonded devices are directly connected to (i.e., where the other end of
1912
the cable plugs into). This may be an actual dedicated switch device,
1913
or it may be another regular system (e.g., another computer running
1914
Linux),
1916
The active-backup, balance-tlb and balance-alb modes do not
1917
require any specific configuration of the switch.
1919
The 802.3ad mode requires that the switch have the appropriate
1920
ports configured as an 802.3ad aggregation. The precise method used
1921
to configure this varies from switch to switch, but, for example, a
1922
Cisco 3550 series switch requires that the appropriate ports first be
1923
grouped together in a single etherchannel instance, then that
1924
etherchannel is set to mode "lacp" to enable 802.3ad (instead of
1925
standard EtherChannel).
1927
The balance-rr, balance-xor and broadcast modes generally
1928
require that the switch have the appropriate ports grouped together.
1929
The nomenclature for such a group differs between switches, it may be
1930
called an "etherchannel" (as in the Cisco example, above), a "trunk
1931
group" or some other similar variation. For these modes, each switch
1932
will also have its own configuration options for the switch's transmit
1933
policy to the bond. Typical choices include XOR of either the MAC or
1934
IP addresses. The transmit policy of the two peers does not need to
1935
match. For these three modes, the bonding mode really selects a
1936
transmit policy for an EtherChannel group; all three will interoperate
1937
with another EtherChannel group.
1940
6. 802.1q VLAN Support
1941
======================
1943
It is possible to configure VLAN devices over a bond interface
1944
using the 8021q driver. However, only packets coming from the 8021q
1945
driver and passing through bonding will be tagged by default. Self
1946
generated packets, for example, bonding's learning packets or ARP
1947
packets generated by either ALB mode or the ARP monitor mechanism, are
1948
tagged internally by bonding itself. As a result, bonding must
1949
"learn" the VLAN IDs configured above it, and use those IDs to tag
1950
self generated packets.
1952
For reasons of simplicity, and to support the use of adapters
1953
that can do VLAN hardware acceleration offloading, the bonding
1954
interface declares itself as fully hardware offloading capable, it gets
1955
the add_vid/kill_vid notifications to gather the necessary
1956
information, and it propagates those actions to the slaves. In case
1957
of mixed adapter types, hardware accelerated tagged packets that
1958
should go through an adapter that is not offloading capable are
1959
"un-accelerated" by the bonding driver so the VLAN tag sits in the
1960
regular location.
1962
VLAN interfaces *must* be added on top of a bonding interface
1963
only after enslaving at least one slave. The bonding interface has a
1964
hardware address of 00:00:00:00:00:00 until the first slave is added.
1965
If the VLAN interface is created prior to the first enslavement, it
1966
would pick up the all-zeroes hardware address. Once the first slave
1967
is attached to the bond, the bond device itself will pick up the
1968
slave's hardware address, which is then available for the VLAN device.
1970
Also, be aware that a similar problem can occur if all slaves
1971
are released from a bond that still has one or more VLAN interfaces on
1972
top of it. When a new slave is added, the bonding interface will
1973
obtain its hardware address from the first slave, which might not
1974
match the hardware address of the VLAN interfaces (which was
1975
ultimately copied from an earlier slave).
1977
There are two methods to ensure that the VLAN device operates
1978
with the correct hardware address if all slaves are removed from a
1979
bond interface:
1981
1. Remove all VLAN interfaces then recreate them
1983
2. Set the bonding interface's hardware address so that it
1984
matches the hardware address of the VLAN interfaces.
1986
Note that changing a VLAN interface's HW address would set the
1987
underlying device -- i.e. the bonding interface -- to promiscuous
1988
mode, which might not be what you want.
1991
7. Link Monitoring
1992
==================
1994
The bonding driver at present supports two schemes for
1995
monitoring a slave device's link state: the ARP monitor and the MII
1996
monitor.
1998
At the present time, due to implementation restrictions in the
1999
bonding driver itself, it is not possible to enable both ARP and MII
2000
monitoring simultaneously.
2002
7.1 ARP Monitor Operation
2003
-------------------------
2005
The ARP monitor operates as its name suggests: it sends ARP
2006
queries to one or more designated peer systems on the network, and
2007
uses the response as an indication that the link is operating. This
2008
gives some assurance that traffic is actually flowing to and from one
2009
or more peers on the local network.
2011
7.2 Configuring Multiple ARP Targets
2012
------------------------------------
2014
While ARP monitoring can be done with just one target, it can
2015
be useful in a High Availability setup to have several targets to
2016
monitor. In the case of just one target, the target itself may go
2017
down or have a problem making it unresponsive to ARP requests. Having
2018
an additional target (or several) increases the reliability of the ARP
2019
monitoring.
2021
Multiple ARP targets must be separated by commas as follows::
2023
# example options for ARP monitoring with three targets
2024
alias bond0 bonding
2025
options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
2027
For just a single target the options would resemble::
2029
# example options for ARP monitoring with one target
2030
alias bond0 bonding
2031
options bond0 arp_interval=60 arp_ip_target=192.168.0.100
2034
7.3 MII Monitor Operation
2035
-------------------------
2037
The MII monitor monitors only the carrier state of the local
2038
network interface. It accomplishes this in one of three ways: by
2039
depending upon the device driver to maintain its carrier state, by
2040
querying the device's MII registers, or by making an ethtool query to
2041
the device.
2043
The MII monitor relies on the driver for carrier state information (via
2044
the netif_carrier subsystem).
2046
8. Potential Sources of Trouble
2047
===============================
2049
8.1 Adventures in Routing
2050
-------------------------
2052
When bonding is configured, it is important that the slave
2053
devices not have routes that supersede routes of the master (or,
2054
generally, not have routes at all). For example, suppose the bonding
2055
device bond0 has two slaves, eth0 and eth1, and the routing table is
2056
as follows::
2058
Kernel IP routing table
2059
Destination Gateway Genmask Flags MSS Window irtt Iface
2060
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0
2061
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1
2062
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0
2063
127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo
2065
This routing configuration will likely still update the
2066
receive/transmit times in the driver (needed by the ARP monitor), but
2067
may bypass the bonding driver (because outgoing traffic to, in this
2068
case, another host on network 10 would use eth0 or eth1 before bond0).
2070
The ARP monitor (and ARP itself) may become confused by this
2071
configuration, because ARP requests (generated by the ARP monitor)
2072
will be sent on one interface (bond0), but the corresponding reply
2073
will arrive on a different interface (eth0). This reply looks to ARP
2074
as an unsolicited ARP reply (because ARP matches replies on an
2075
interface basis), and is discarded. The MII monitor is not affected
2076
by the state of the routing table.
2078
The solution here is simply to ensure that slaves do not have
2079
routes of their own, and if for some reason they must, those routes do
2080
not supersede routes of their master. This should generally be the
2081
case, but unusual configurations or errant manual or automatic static
2082
route additions may cause trouble.
2084
8.2 Ethernet Device Renaming
2085
----------------------------
2087
On systems with network configuration scripts that do not
2088
associate physical devices directly with network interface names (so
2089
that the same physical device always has the same "ethX" name), it may
2090
be necessary to add some special logic to config files in
2091
/etc/modprobe.d/.
2093
For example, given a modules.conf containing the following::
2095
alias bond0 bonding
2096
options bond0 mode=some-mode miimon=50
2097
alias eth0 tg3
2098
alias eth1 tg3
2099
alias eth2 e1000
2100
alias eth3 e1000
2102
If neither eth0 and eth1 are slaves to bond0, then when the
2103
bond0 interface comes up, the devices may end up reordered. This
2104
happens because bonding is loaded first, then its slave device's
2105
drivers are loaded next. Since no other drivers have been loaded,
2106
when the e1000 driver loads, it will receive eth0 and eth1 for its
2107
devices, but the bonding configuration tries to enslave eth2 and eth3
2108
(which may later be assigned to the tg3 devices).
2110
Adding the following::
2112
add above bonding e1000 tg3
2114
causes modprobe to load e1000 then tg3, in that order, when
2115
bonding is loaded. This command is fully documented in the
2116
modules.conf manual page.
2118
On systems utilizing modprobe an equivalent problem can occur.
2119
In this case, the following can be added to config files in
2120
/etc/modprobe.d/ as::
2122
softdep bonding pre: tg3 e1000
2124
This will load tg3 and e1000 modules before loading the bonding one.
2125
Full documentation on this can be found in the modprobe.d and modprobe
2126
manual pages.
2128
9. SNMP agents
2129
===============
2131
If running SNMP agents, the bonding driver should be loaded
2132
before any network drivers participating in a bond. This requirement
2133
is due to the interface index (ipAdEntIfIndex) being associated to
2134
the first interface found with a given IP address. That is, there is
2135
only one ipAdEntIfIndex for each IP address. For example, if eth0 and
2136
eth1 are slaves of bond0 and the driver for eth0 is loaded before the
2137
bonding driver, the interface for the IP address will be associated
2138
with the eth0 interface. This configuration is shown below, the IP
2139
address 192.168.1.1 has an interface index of 2 which indexes to eth0
2140
in the ifDescr table (ifDescr.2).
2142
::
2144
interfaces.ifTable.ifEntry.ifDescr.1 = lo
2145
interfaces.ifTable.ifEntry.ifDescr.2 = eth0
2146
interfaces.ifTable.ifEntry.ifDescr.3 = eth1
2147
interfaces.ifTable.ifEntry.ifDescr.4 = eth2
2148
interfaces.ifTable.ifEntry.ifDescr.5 = eth3
2149
interfaces.ifTable.ifEntry.ifDescr.6 = bond0
2150
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
2151
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
2152
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
2153
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
2155
This problem is avoided by loading the bonding driver before
2156
any network drivers participating in a bond. Below is an example of
2157
loading the bonding driver first, the IP address 192.168.1.1 is
2158
correctly associated with ifDescr.2.
2160
interfaces.ifTable.ifEntry.ifDescr.1 = lo
2161
interfaces.ifTable.ifEntry.ifDescr.2 = bond0
2162
interfaces.ifTable.ifEntry.ifDescr.3 = eth0
2163
interfaces.ifTable.ifEntry.ifDescr.4 = eth1
2164
interfaces.ifTable.ifEntry.ifDescr.5 = eth2
2165
interfaces.ifTable.ifEntry.ifDescr.6 = eth3
2166
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
2167
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
2168
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
2169
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
2171
While some distributions may not report the interface name in
2172
ifDescr, the association between the IP address and IfIndex remains
2173
and SNMP functions such as Interface_Scan_Next will report that
2174
association.
2176
10. Promiscuous mode
2177
====================
2179
When running network monitoring tools, e.g., tcpdump, it is
2180
common to enable promiscuous mode on the device, so that all traffic
2181
is seen (instead of seeing only traffic destined for the local host).
2182
The bonding driver handles promiscuous mode changes to the bonding
2183
master device (e.g., bond0), and propagates the setting to the slave
2184
devices.
2186
For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
2187
the promiscuous mode setting is propagated to all slaves.
2189
For the active-backup, balance-tlb and balance-alb modes, the
2190
promiscuous mode setting is propagated only to the active slave.
2192
For balance-tlb mode, the active slave is the slave currently
2193
receiving inbound traffic.
2195
For balance-alb mode, the active slave is the slave used as a
2196
"primary." This slave is used for mode-specific control traffic, for
2197
sending to peers that are unassigned or if the load is unbalanced.
2199
For the active-backup, balance-tlb and balance-alb modes, when
2200
the active slave changes (e.g., due to a link failure), the
2201
promiscuous setting will be propagated to the new active slave.
2203
11. Configuring Bonding for High Availability
2204
=============================================
2206
High Availability refers to configurations that provide
2207
maximum network availability by having redundant or backup devices,
2208
links or switches between the host and the rest of the world. The
2209
goal is to provide the maximum availability of network connectivity
2210
(i.e., the network always works), even though other configurations
2211
could provide higher throughput.
2213
11.1 High Availability in a Single Switch Topology
2214
--------------------------------------------------
2216
If two hosts (or a host and a single switch) are directly
2217
connected via multiple physical links, then there is no availability
2218
penalty to optimizing for maximum bandwidth. In this case, there is
2219
only one switch (or peer), so if it fails, there is no alternative
2220
access to fail over to. Additionally, the bonding load balance modes
2221
support link monitoring of their members, so if individual links fail,
2222
the load will be rebalanced across the remaining devices.
2224
See Section 12, "Configuring Bonding for Maximum Throughput"
2225
for information on configuring bonding with one peer device.
2227
11.2 High Availability in a Multiple Switch Topology
2228
----------------------------------------------------
2230
With multiple switches, the configuration of bonding and the
2231
network changes dramatically. In multiple switch topologies, there is
2232
a trade off between network availability and usable bandwidth.
2234
Below is a sample network, configured to maximize the
2235
availability of the network::
2237
| |
2238
|port3 port3|
2239
+-----+----+ +-----+----+
2240
| |port2 ISL port2| |
2241
| switch A +--------------------------+ switch B |
2242
| | | |
2243
+-----+----+ +-----++---+
2244
|port1 port1|
2245
| +-------+ |
2246
+-------------+ host1 +---------------+
2247
eth0 +-------+ eth1
2249
In this configuration, there is a link between the two
2250
switches (ISL, or inter switch link), and multiple ports connecting to
2251
the outside world ("port3" on each switch). There is no technical
2252
reason that this could not be extended to a third switch.
2254
11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
2255
-------------------------------------------------------------
2257
In a topology such as the example above, the active-backup and
2258
broadcast modes are the only useful bonding modes when optimizing for
2259
availability; the other modes require all links to terminate on the
2260
same peer for them to behave rationally.
2262
active-backup:
2263
This is generally the preferred mode, particularly if
2264
the switches have an ISL and play together well. If the
2265
network configuration is such that one switch is specifically
2266
a backup switch (e.g., has lower capacity, higher cost, etc),
2267
then the primary option can be used to ensure that the
2268
preferred link is always used when it is available.
2270
broadcast:
2271
This mode is really a special purpose mode, and is suitable
2272
only for very specific needs. For example, if the two
2273
switches are not connected (no ISL), and the networks beyond
2274
them are totally independent. In this case, if it is
2275
necessary for some specific one-way traffic to reach both
2276
independent networks, then the broadcast mode may be suitable.
2278
11.2.2 HA Link Monitoring Selection for Multiple Switch Topology
2279
----------------------------------------------------------------
2281
The choice of link monitoring ultimately depends upon your
2282
switch. If the switch can reliably fail ports in response to other
2283
failures, then either the MII or ARP monitors should work. For
2284
example, in the above example, if the "port3" link fails at the remote
2285
end, the MII monitor has no direct means to detect this. The ARP
2286
monitor could be configured with a target at the remote end of port3,
2287
thus detecting that failure without switch support.
2289
In general, however, in a multiple switch topology, the ARP
2290
monitor can provide a higher level of reliability in detecting end to
2291
end connectivity failures (which may be caused by the failure of any
2292
individual component to pass traffic for any reason). Additionally,
2293
the ARP monitor should be configured with multiple targets (at least
2294
one for each switch in the network). This will ensure that,
2295
regardless of which switch is active, the ARP monitor has a suitable
2296
target to query.
2298
Note, also, that of late many switches now support a functionality
2299
generally referred to as "trunk failover." This is a feature of the
2300
switch that causes the link state of a particular switch port to be set
2301
down (or up) when the state of another switch port goes down (or up).
2302
Its purpose is to propagate link failures from logically "exterior" ports
2303
to the logically "interior" ports that bonding is able to monitor via
2304
miimon. Availability and configuration for trunk failover varies by
2305
switch, but this can be a viable alternative to the ARP monitor when using
2306
suitable switches.
2308
12. Configuring Bonding for Maximum Throughput
2309
==============================================
2311
12.1 Maximizing Throughput in a Single Switch Topology
2312
------------------------------------------------------
2314
In a single switch configuration, the best method to maximize
2315
throughput depends upon the application and network environment. The
2316
various load balancing modes each have strengths and weaknesses in
2317
different environments, as detailed below.
2319
For this discussion, we will break down the topologies into
2320
two categories. Depending upon the destination of most traffic, we
2321
categorize them into either "gatewayed" or "local" configurations.
2323
In a gatewayed configuration, the "switch" is acting primarily
2324
as a router, and the majority of traffic passes through this router to
2325
other networks. An example would be the following::
2328
+----------+ +----------+
2329
| |eth0 port1| | to other networks
2330
| Host A +---------------------+ router +------------------->
2331
| +---------------------+ | Hosts B and C are out
2332
| |eth1 port2| | here somewhere
2333
+----------+ +----------+
2335
The router may be a dedicated router device, or another host
2336
acting as a gateway. For our discussion, the important point is that
2337
the majority of traffic from Host A will pass through the router to
2338
some other network before reaching its final destination.
2340
In a gatewayed network configuration, although Host A may
2341
communicate with many other systems, all of its traffic will be sent
2342
and received via one other peer on the local network, the router.
2344
Note that the case of two systems connected directly via
2345
multiple physical links is, for purposes of configuring bonding, the
2346
same as a gatewayed configuration. In that case, it happens that all
2347
traffic is destined for the "gateway" itself, not some other network
2348
beyond the gateway.
2350
In a local configuration, the "switch" is acting primarily as
2351
a switch, and the majority of traffic passes through this switch to
2352
reach other stations on the same network. An example would be the
2353
following::
2355
+----------+ +----------+ +--------+
2356
| |eth0 port1| +-------+ Host B |
2357
| Host A +------------+ switch |port3 +--------+
2358
| +------------+ | +--------+
2359
| |eth1 port2| +------------------+ Host C |
2360
+----------+ +----------+port4 +--------+
2363
Again, the switch may be a dedicated switch device, or another
2364
host acting as a gateway. For our discussion, the important point is
2365
that the majority of traffic from Host A is destined for other hosts
2366
on the same local network (Hosts B and C in the above example).
2368
In summary, in a gatewayed configuration, traffic to and from
2369
the bonded device will be to the same MAC level peer on the network
2370
(the gateway itself, i.e., the router), regardless of its final
2371
destination. In a local configuration, traffic flows directly to and
2372
from the final destinations, thus, each destination (Host B, Host C)
2373
will be addressed directly by their individual MAC addresses.
2375
This distinction between a gatewayed and a local network
2376
configuration is important because many of the load balancing modes
2377
available use the MAC addresses of the local network source and
2378
destination to make load balancing decisions. The behavior of each
2379
mode is described below.
2382
12.1.1 MT Bonding Mode Selection for Single Switch Topology
2383
-----------------------------------------------------------
2385
This configuration is the easiest to set up and to understand,
2386
although you will have to decide which bonding mode best suits your
2387
needs. The trade offs for each mode are detailed below:
2389
balance-rr:
2390
This mode is the only mode that will permit a single
2391
TCP/IP connection to stripe traffic across multiple
2392
interfaces. It is therefore the only mode that will allow a
2393
single TCP/IP stream to utilize more than one interface's
2394
worth of throughput. This comes at a cost, however: the
2395
striping generally results in peer systems receiving packets out
2396
of order, causing TCP/IP's congestion control system to kick
2397
in, often by retransmitting segments.
2399
It is possible to adjust TCP/IP's congestion limits by
2400
altering the net.ipv4.tcp_reordering sysctl parameter. The
2401
usual default value is 3. But keep in mind TCP stack is able
2402
to automatically increase this when it detects reorders.
2404
Note that the fraction of packets that will be delivered out of
2405
order is highly variable, and is unlikely to be zero. The level
2406
of reordering depends upon a variety of factors, including the
2407
networking interfaces, the switch, and the topology of the
2408
configuration. Speaking in general terms, higher speed network
2409
cards produce more reordering (due to factors such as packet
2410
coalescing), and a "many to many" topology will reorder at a
2411
higher rate than a "many slow to one fast" configuration.
2413
Many switches do not support any modes that stripe traffic
2414
(instead choosing a port based upon IP or MAC level addresses);
2415
for those devices, traffic for a particular connection flowing
2416
through the switch to a balance-rr bond will not utilize greater
2417
than one interface's worth of bandwidth.
2419
If you are utilizing protocols other than TCP/IP, UDP for
2420
example, and your application can tolerate out of order
2421
delivery, then this mode can allow for single stream datagram
2422
performance that scales near linearly as interfaces are added
2423
to the bond.
2425
This mode requires the switch to have the appropriate ports
2426
configured for "etherchannel" or "trunking."
2428
active-backup:
2429
There is not much advantage in this network topology to
2430
the active-backup mode, as the inactive backup devices are all
2431
connected to the same peer as the primary. In this case, a
2432
load balancing mode (with link monitoring) will provide the
2433
same level of network availability, but with increased
2434
available bandwidth. On the plus side, active-backup mode
2435
does not require any configuration of the switch, so it may
2436
have value if the hardware available does not support any of
2437
the load balance modes.
2439
balance-xor:
2440
This mode will limit traffic such that packets destined
2441
for specific peers will always be sent over the same
2442
interface. Since the destination is determined by the MAC
2443
addresses involved, this mode works best in a "local" network
2444
configuration (as described above), with destinations all on
2445
the same local network. This mode is likely to be suboptimal
2446
if all your traffic is passed through a single router (i.e., a
2447
"gatewayed" network configuration, as described above).
2449
As with balance-rr, the switch ports need to be configured for
2450
"etherchannel" or "trunking."
2452
broadcast:
2453
Like active-backup, there is not much advantage to this
2454
mode in this type of network topology.
2456
802.3ad:
2457
This mode can be a good choice for this type of network
2458
topology. The 802.3ad mode is an IEEE standard, so all peers
2459
that implement 802.3ad should interoperate well. The 802.3ad
2460
protocol includes automatic configuration of the aggregates,
2461
so minimal manual configuration of the switch is needed
2462
(typically only to designate that some set of devices is
2463
available for 802.3ad). The 802.3ad standard also mandates
2464
that frames be delivered in order (within certain limits), so
2465
in general single connections will not see misordering of
2466
packets. The 802.3ad mode does have some drawbacks: the
2467
standard mandates that all devices in the aggregate operate at
2468
the same speed and duplex. Also, as with all bonding load
2469
balance modes other than balance-rr, no single connection will
2470
be able to utilize more than a single interface's worth of
2471
bandwidth.
2473
Additionally, the linux bonding 802.3ad implementation
2474
distributes traffic by peer (using an XOR of MAC addresses
2475
and packet type ID), so in a "gatewayed" configuration, all
2476
outgoing traffic will generally use the same device. Incoming
2477
traffic may also end up on a single device, but that is
2478
dependent upon the balancing policy of the peer's 802.3ad
2479
implementation. In a "local" configuration, traffic will be
2480
distributed across the devices in the bond.
2482
Finally, the 802.3ad mode mandates the use of the MII monitor,
2483
therefore, the ARP monitor is not available in this mode.
2485
balance-tlb:
2486
The balance-tlb mode balances outgoing traffic by peer.
2487
Since the balancing is done according to MAC address, in a
2488
"gatewayed" configuration (as described above), this mode will
2489
send all traffic across a single device. However, in a
2490
"local" network configuration, this mode balances multiple
2491
local network peers across devices in a vaguely intelligent
2492
manner (not a simple XOR as in balance-xor or 802.3ad mode),
2493
so that mathematically unlucky MAC addresses (i.e., ones that
2494
XOR to the same value) will not all "bunch up" on a single
2495
interface.
2497
Unlike 802.3ad, interfaces may be of differing speeds, and no
2498
special switch configuration is required. On the down side,
2499
in this mode all incoming traffic arrives over a single
2500
interface, this mode requires certain ethtool support in the
2501
network device driver of the slave interfaces, and the ARP
2502
monitor is not available.
2504
balance-alb:
2505
This mode is everything that balance-tlb is, and more.
2506
It has all of the features (and restrictions) of balance-tlb,
2507
and will also balance incoming traffic from local network
2508
peers (as described in the Bonding Module Options section,
2509
above).
2511
The only additional down side to this mode is that the network
2512
device driver must support changing the hardware address while
2513
the device is open.
2515
12.1.2 MT Link Monitoring for Single Switch Topology
2516
----------------------------------------------------
2518
The choice of link monitoring may largely depend upon which
2519
mode you choose to use. The more advanced load balancing modes do not
2520
support the use of the ARP monitor, and are thus restricted to using
2521
the MII monitor (which does not provide as high a level of end to end
2522
assurance as the ARP monitor).
2524
12.2 Maximum Throughput in a Multiple Switch Topology
2525
-----------------------------------------------------
2527
Multiple switches may be utilized to optimize for throughput
2528
when they are configured in parallel as part of an isolated network
2529
between two or more systems, for example::
2531
+-----------+
2532
| Host A |
2533
+-+---+---+-+
2534
| | |
2535
+--------+ | +---------+
2536
| | |
2537
+------+---+ +-----+----+ +-----+----+
2538
| Switch A | | Switch B | | Switch C |
2539
+------+---+ +-----+----+ +-----+----+
2540
| | |
2541
+--------+ | +---------+
2542
| | |
2543
+-+---+---+-+
2544
| Host B |
2545
+-----------+
2547
In this configuration, the switches are isolated from one
2548
another. One reason to employ a topology such as this is for an
2549
isolated network with many hosts (a cluster configured for high
2550
performance, for example), using multiple smaller switches can be more
2551
cost effective than a single larger switch, e.g., on a network with 24
2552
hosts, three 24 port switches can be significantly less expensive than
2553
a single 72 port switch.
2555
If access beyond the network is required, an individual host
2556
can be equipped with an additional network device connected to an
2557
external network; this host then additionally acts as a gateway.
2559
12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
2560
-------------------------------------------------------------
2562
In actual practice, the bonding mode typically employed in
2563
configurations of this type is balance-rr. Historically, in this
2564
network configuration, the usual caveats about out of order packet
2565
delivery are mitigated by the use of network adapters that do not do
2566
any kind of packet coalescing (via the use of NAPI, or because the
2567
device itself does not generate interrupts until some number of
2568
packets has arrived). When employed in this fashion, the balance-rr
2569
mode allows individual connections between two hosts to effectively
2570
utilize greater than one interface's bandwidth.
2572
12.2.2 MT Link Monitoring for Multiple Switch Topology
2573
------------------------------------------------------
2575
Again, in actual practice, the MII monitor is most often used
2576
in this configuration, as performance is given preference over
2577
availability. The ARP monitor will function in this topology, but its
2578
advantages over the MII monitor are mitigated by the volume of probes
2579
needed as the number of systems involved grows (remember that each
2580
host in the network is configured with bonding).
2582
13. Switch Behavior Issues
2583
==========================
2585
13.1 Link Establishment and Failover Delays
2586
-------------------------------------------
2588
Some switches exhibit undesirable behavior with regard to the
2589
timing of link up and down reporting by the switch.
2591
First, when a link comes up, some switches may indicate that
2592
the link is up (carrier available), but not pass traffic over the
2593
interface for some period of time. This delay is typically due to
2594
some type of autonegotiation or routing protocol, but may also occur
2595
during switch initialization (e.g., during recovery after a switch
2596
failure). If you find this to be a problem, specify an appropriate
2597
value to the updelay bonding module option to delay the use of the
2598
relevant interface(s).
2600
Second, some switches may "bounce" the link state one or more
2601
times while a link is changing state. This occurs most commonly while
2602
the switch is initializing. Again, an appropriate updelay value may
2603
help.
2605
Note that when a bonding interface has no active links, the
2606
driver will immediately reuse the first link that goes up, even if the
2607
updelay parameter has been specified (the updelay is ignored in this
2608
case). If there are slave interfaces waiting for the updelay timeout
2609
to expire, the interface that first went into that state will be
2610
immediately reused. This reduces down time of the network if the
2611
value of updelay has been overestimated, and since this occurs only in
2612
cases with no connectivity, there is no additional penalty for
2613
ignoring the updelay.
2615
In addition to the concerns about switch timings, if your
2616
switches take a long time to go into backup mode, it may be desirable
2617
to not activate a backup interface immediately after a link goes down.
2618
Failover may be delayed via the downdelay bonding module option.
2620
13.2 Duplicated Incoming Packets
2621
--------------------------------
2623
NOTE: Starting with version 3.0.2, the bonding driver has logic to
2624
suppress duplicate packets, which should largely eliminate this problem.
2625
The following description is kept for reference.
2627
It is not uncommon to observe a short burst of duplicated
2628
traffic when the bonding device is first used, or after it has been
2629
idle for some period of time. This is most easily observed by issuing
2630
a "ping" to some other host on the network, and noticing that the
2631
output from ping flags duplicates (typically one per slave).
2633
For example, on a bond in active-backup mode with five slaves
2634
all connected to one switch, the output may appear as follows::
2636
# ping -n 10.0.4.2
2637
PING 10.0.4.2 (10.0.4.2) from 10.0.3.10 : 56(84) bytes of data.
2638
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.7 ms
2639
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2640
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2641
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2642
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2643
64 bytes from 10.0.4.2: icmp_seq=2 ttl=64 time=0.216 ms
2644
64 bytes from 10.0.4.2: icmp_seq=3 ttl=64 time=0.267 ms
2645
64 bytes from 10.0.4.2: icmp_seq=4 ttl=64 time=0.222 ms
2647
This is not due to an error in the bonding driver, rather, it
2648
is a side effect of how many switches update their MAC forwarding
2649
tables. Initially, the switch does not associate the MAC address in
2650
the packet with a particular switch port, and so it may send the
2651
traffic to all ports until its MAC forwarding table is updated. Since
2652
the interfaces attached to the bond may occupy multiple ports on a
2653
single switch, when the switch (temporarily) floods the traffic to all
2654
ports, the bond device receives multiple copies of the same packet
2655
(one per slave device).
2657
The duplicated packet behavior is switch dependent, some
2658
switches exhibit this, and some do not. On switches that display this
2659
behavior, it can be induced by clearing the MAC forwarding table (on
2660
most Cisco switches, the privileged command "clear mac address-table
2661
dynamic" will accomplish this).
2663
14. Hardware Specific Considerations
2664
====================================
2666
This section contains additional information for configuring
2667
bonding on specific hardware platforms, or for interfacing bonding
2668
with particular switches or other devices.
2670
14.1 IBM BladeCenter
2671
--------------------
2673
This applies to the JS20 and similar systems.
2675
On the JS20 blades, the bonding driver supports only
2676
balance-rr, active-backup, balance-tlb and balance-alb modes. This is
2677
largely due to the network topology inside the BladeCenter, detailed
2678
below.
2680
JS20 network adapter information
2681
--------------------------------
2683
All JS20s come with two Broadcom Gigabit Ethernet ports
2684
integrated on the planar (that's "motherboard" in IBM-speak). In the
2685
BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to
2686
I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2.
2687
An add-on Broadcom daughter card can be installed on a JS20 to provide
2688
two more Gigabit Ethernet ports. These ports, eth2 and eth3, are
2689
wired to I/O Modules 3 and 4, respectively.
2691
Each I/O Module may contain either a switch or a passthrough
2692
module (which allows ports to be directly connected to an external
2693
switch). Some bonding modes require a specific BladeCenter internal
2694
network topology in order to function; these are detailed below.
2696
Additional BladeCenter-specific networking information can be
2697
found in two IBM Redbooks (www.ibm.com/redbooks):
2699
- "IBM eServer BladeCenter Networking Options"
2700
- "IBM eServer BladeCenter Layer 2-7 Network Switching"
2702
BladeCenter networking configuration
2703
------------------------------------
2705
Because a BladeCenter can be configured in a very large number
2706
of ways, this discussion will be confined to describing basic
2707
configurations.
2709
Normally, Ethernet Switch Modules (ESMs) are used in I/O
2710
modules 1 and 2. In this configuration, the eth0 and eth1 ports of a
2711
JS20 will be connected to different internal switches (in the
2712
respective I/O modules).
2714
A passthrough module (OPM or CPM, optical or copper,
2715
passthrough module) connects the I/O module directly to an external
2716
switch. By using PMs in I/O module #1 and #2, the eth0 and eth1
2717
interfaces of a JS20 can be redirected to the outside world and
2718
connected to a common external switch.
2720
Depending upon the mix of ESMs and PMs, the network will
2721
appear to bonding as either a single switch topology (all PMs) or as a
2722
multiple switch topology (one or more ESMs, zero or more PMs). It is
2723
also possible to connect ESMs together, resulting in a configuration
2724
much like the example in "High Availability in a Multiple Switch
2725
Topology," above.
2727
Requirements for specific modes
2728
-------------------------------
2730
The balance-rr mode requires the use of passthrough modules
2731
for devices in the bond, all connected to an common external switch.
2732
That switch must be configured for "etherchannel" or "trunking" on the
2733
appropriate ports, as is usual for balance-rr.
2735
The balance-alb and balance-tlb modes will function with
2736
either switch modules or passthrough modules (or a mix). The only
2737
specific requirement for these modes is that all network interfaces
2738
must be able to reach all destinations for traffic sent over the
2739
bonding device (i.e., the network must converge at some point outside
2740
the BladeCenter).
2742
The active-backup mode has no additional requirements.
2744
Link monitoring issues
2745
----------------------
2747
When an Ethernet Switch Module is in place, only the ARP
2748
monitor will reliably detect link loss to an external switch. This is
2749
nothing unusual, but examination of the BladeCenter cabinet would
2750
suggest that the "external" network ports are the ethernet ports for
2751
the system, when it fact there is a switch between these "external"
2752
ports and the devices on the JS20 system itself. The MII monitor is
2753
only able to detect link failures between the ESM and the JS20 system.
2755
When a passthrough module is in place, the MII monitor does
2756
detect failures to the "external" port, which is then directly
2757
connected to the JS20 system.
2759
Other concerns
2760
--------------
2762
The Serial Over LAN (SoL) link is established over the primary
2763
ethernet (eth0) only, therefore, any loss of link to eth0 will result
2764
in losing your SoL connection. It will not fail over with other
2765
network traffic, as the SoL system is beyond the control of the
2766
bonding driver.
2768
It may be desirable to disable spanning tree on the switch
2769
(either the internal Ethernet Switch Module, or an external switch) to
2770
avoid fail-over delay issues when using bonding.
2773
15. Frequently Asked Questions
2774
==============================
2776
1. Is it SMP safe?
2777
-------------------
2779
Yes. The old 2.0.xx channel bonding patch was not SMP safe.
2780
The new driver was designed to be SMP safe from the start.
2782
2. What type of cards will work with it?
2783
-----------------------------------------
2785
Any Ethernet type cards (you can even mix cards - a Intel
2786
EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes,
2787
devices need not be of the same speed.
2789
Starting with version 3.2.1, bonding also supports Infiniband
2790
slaves in active-backup mode.
2792
3. How many bonding devices can I have?
2793
----------------------------------------
2795
There is no limit.
2797
4. How many slaves can a bonding device have?
2798
----------------------------------------------
2800
This is limited only by the number of network interfaces Linux
2801
supports and/or the number of network cards you can place in your
2802
system.
2804
5. What happens when a slave link dies?
2805
----------------------------------------
2807
If link monitoring is enabled, then the failing device will be
2808
disabled. The active-backup mode will fail over to a backup link, and
2809
other modes will ignore the failed link. The link will continue to be
2810
monitored, and should it recover, it will rejoin the bond (in whatever
2811
manner is appropriate for the mode). See the sections on High
2812
Availability and the documentation for each mode for additional
2813
information.
2815
Link monitoring can be enabled via either the miimon or
2816
arp_interval parameters (described in the module parameters section,
2817
above). In general, miimon monitors the carrier state as sensed by
2818
the underlying network device, and the arp monitor (arp_interval)
2819
monitors connectivity to another host on the local network.
2821
If no link monitoring is configured, the bonding driver will
2822
be unable to detect link failures, and will assume that all links are
2823
always available. This will likely result in lost packets, and a
2824
resulting degradation of performance. The precise performance loss
2825
depends upon the bonding mode and network configuration.
2827
6. Can bonding be used for High Availability?
2828
----------------------------------------------
2830
Yes. See the section on High Availability for details.
2832
7. Which switches/systems does it work with?
2833
---------------------------------------------
2835
The full answer to this depends upon the desired mode.
2837
In the basic balance modes (balance-rr and balance-xor), it
2838
works with any system that supports etherchannel (also called
2839
trunking). Most managed switches currently available have such
2840
support, and many unmanaged switches as well.
2842
The advanced balance modes (balance-tlb and balance-alb) do
2843
not have special switch requirements, but do need device drivers that
2844
support specific features (described in the appropriate section under
2845
module parameters, above).
2847
In 802.3ad mode, it works with systems that support IEEE
2848
802.3ad Dynamic Link Aggregation. Most managed and many unmanaged
2849
switches currently available support 802.3ad.
2851
The active-backup mode should work with any Layer-II switch.
2853
8. Where does a bonding device get its MAC address from?
2854
---------------------------------------------------------
2856
When using slave devices that have fixed MAC addresses, or when
2857
the fail_over_mac option is enabled, the bonding device's MAC address is
2858
the MAC address of the active slave.
2860
For other configurations, if not explicitly configured (with
2861
ifconfig or ip link), the MAC address of the bonding device is taken from
2862
its first slave device. This MAC address is then passed to all following
2863
slaves and remains persistent (even if the first slave is removed) until
2864
the bonding device is brought down or reconfigured.
2866
If you wish to change the MAC address, you can set it with
2867
ifconfig or ip link::
2869
# ifconfig bond0 hw ether 00:11:22:33:44:55
2871
# ip link set bond0 address 66:77:88:99:aa:bb
2873
The MAC address can be also changed by bringing down/up the
2874
device and then changing its slaves (or their order)::
2876
# ifconfig bond0 down ; modprobe -r bonding
2877
# ifconfig bond0 .... up
2878
# ifenslave bond0 eth...
2880
This method will automatically take the address from the next
2881
slave that is added.
2883
To restore your slaves' MAC addresses, you need to detach them
2884
from the bond (``ifenslave -d bond0 eth0``). The bonding driver will
2885
then restore the MAC addresses that the slaves had before they were
2886
enslaved.
2888
9. What bonding modes support native XDP?
2889
------------------------------------------
2891
* balance-rr (0)
2892
* active-backup (1)
2893
* balance-xor (2)
2894
* 802.3ad (4)
2896
Note that the vlan+srcmac hash policy does not support native XDP.
2897
For other bonding modes, the XDP program must be loaded with generic mode.
2899
16. Resources and Links
2900
=======================
2902
The latest version of the bonding driver can be found in the latest
2903
version of the linux kernel, found on http://kernel.org
2905
The latest version of this document can be found in the latest kernel
2906
source (named Documentation/networking/bonding.rst).
2908
Discussions regarding the development of the bonding driver take place
2909
on the main Linux network mailing list, hosted at vger.kernel.org. The list
2910
address is:
2914
The administrative interface (to subscribe or unsubscribe) can
2915
be found at:
2917
http://vger.kernel.org/vger-lists.html#netdev
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서 계보·소개·목차
1-108`.. SPDX-License-Identifier: GPL-2.0`
Linux Ethernet Bonding Driver HOWTO
최종 갱신일은 2011년 4월 27일입니다. 최초 release는 Thomas Davis가 작성했고, 2000년 10월 HA 확장과 수정에는 Willy Tarreau, Constantine Gavrilov, Chad N. Tindel, Janice Girouard, Jay Vosburgh가 참여했습니다. Jay Vosburgh가 2005년 2월 문서를 재구성·갱신했고 Mitch Williams가 2006년 4월 sysfs 정보를 추가했습니다.
소개
Linux bonding driver는 여러 network interface를 하나의 논리적 bonded interface로 합칩니다. 동작은 mode에 따라 달라지며 일반적으로 hot standby 또는 load balancing을 제공하고 link integrity monitoring도 수행할 수 있습니다.
Bonding driver는 원래 kernel 2.0용 Donald Becker의 beowulf patch에서 왔지만 이후 크게 바뀌었습니다. Extreme-linux와 beowulf site의 옛 도구는 이 version과 호환되지 않습니다. 새 driver, user-space tool, 지원 창구는 문서 끝의 link를 참고합니다.
목차는 설치, module option, distro·iproute2·sysfs 구성, 상태 조회, switch와 VLAN, link monitoring, 장애 원인, SNMP·promiscuous mode, HA·최대 처리량 topology, switch 동작, IBM BladeCenter, FAQ와 자료를 차례로 다룹니다.
.. SPDX-License-Identifier: GPL-2.0
===================================
Linux Ethernet Bonding Driver HOWTO
===================================
Latest update: 27 April 2011
Initial release: Thomas Davis <tadavis at lbl.gov>
Corrections, HA extensions: 2000/10/03-15:
- Willy Tarreau <willy at meta-x.org>
- Constantine Gavrilov <const-g at xpert.com>
- Chad N. Tindel <ctindel at ieee dot org>
- Janice Girouard <girouard at us dot ibm dot com>
- Jay Vosburgh <fubar at us dot ibm dot com>
Reorganized and updated Feb 2005 by Jay Vosburgh
Added Sysfs information: 2006/04/24
- Mitch Williams <mitch.a.williams at intel.com>
Introduction
============
The Linux bonding driver provides a method for aggregating
multiple network interfaces into a single logical "bonded" interface.
The behavior of the bonded interfaces depends upon the mode; generally
speaking, modes provide either hot standby or load balancing services.
Additionally, link integrity monitoring may be performed.
The bonding driver originally came from Donald Becker's
beowulf patches for kernel 2.0. It has changed quite a bit since, and
the original tools from extreme-linux and beowulf sites will not work
with this version of the driver.
For new versions of the driver, updated userspace tools, and
who to ask for help, please follow the links at the end of this file.
.. Table of Contents
1. Bonding Driver Installation
2. Bonding Driver Options
3. Configuring Bonding Devices
3.1 Configuration with Sysconfig Support
3.1.1 Using DHCP with Sysconfig
3.1.2 Configuring Multiple Bonds with Sysconfig
3.2 Configuration with Initscripts Support
3.2.1 Using DHCP with Initscripts
3.2.2 Configuring Multiple Bonds with Initscripts
3.3 Configuring Bonding Manually with Ifenslave
3.3.1 Configuring Multiple Bonds Manually
3.4 Configuring Bonding Manually via Sysfs
3.5 Configuration with Interfaces Support
3.6 Overriding Configuration for Special Cases
3.7 Configuring LACP for 802.3ad mode in a more secure way
4. Querying Bonding Configuration
4.1 Bonding Configuration
4.2 Network Configuration
5. Switch Configuration
6. 802.1q VLAN Support
7. Link Monitoring
7.1 ARP Monitor Operation
7.2 Configuring Multiple ARP Targets
7.3 MII Monitor Operation
8. Potential Trouble Sources
8.1 Adventures in Routing
8.2 Ethernet Device Renaming
8.3 Painfully Slow Or No Failed Link Detection By Miimon
9. SNMP agents
10. Promiscuous mode
11. Configuring Bonding for High Availability
11.1 High Availability in a Single Switch Topology
11.2 High Availability in a Multiple Switch Topology
11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
11.2.2 HA Link Monitoring for Multiple Switch Topology
12. Configuring Bonding for Maximum Throughput
12.1 Maximum Throughput in a Single Switch Topology
12.1.1 MT Bonding Mode Selection for Single Switch Topology
12.1.2 MT Link Monitoring for Single Switch Topology
12.2 Maximum Throughput in a Multiple Switch Topology
12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
12.2.2 MT Link Monitoring for Multiple Switch Topology
13. Switch Behavior Issues
13.1 Link Establishment and Failover Delays
13.2 Duplicated Incoming Packets
14. Hardware Specific Considerations
14.1 IBM BladeCenter
15. Frequently Asked Questions
16. Resources and Links
설치와 option 기본 원칙
109-1691. Bonding driver 설치
대부분의 distribution kernel은 bonding driver를 module로 제공합니다. 직접 build해야 한다면 최신 kernel source의 `drivers/net/bonding`을 사용하고 `make menuconfig`, `make xconfig` 또는 `make config`의 Network device support에서 Bonding driver support를 선택한 뒤 kernel과 module을 build·설치합니다.
문서 작성 당시 parameter 전달과 둘 이상의 bond device 구성은 module 방식에서만 가능했으므로 module build를 권장했습니다. 현재 제어는 iproute2(netlink)나 sysfs가 권장되며 옛 `ifenslave` utility는 obsolete입니다.
2. Bonding driver option
Option은 module load parameter 또는 sysfs로 지정합니다. `insmod`·`modprobe` command line, `/etc/modprobe.d/*.conf`, distribution 전용 설정 file에서 제공할 수 있습니다. 생략한 parameter에는 기본값이 적용됩니다.
초기 구성 중에는 별도 window에서 `tail -f /var/log/messages`로 error를 확인하는 것이 좋습니다. Link failure 때 심각한 network 저하를 막으려면 `miimon` 또는 `arp_interval`과 `arp_ip_target`을 반드시 설정해야 합니다. Text option은 이름과 과거 호환용 숫자를 모두 받아 `mode=802.3ad`와 `mode=4`가 같습니다.
1. Bonding Driver Installation
==============================
Most popular distro kernels ship with the bonding driver
already available as a module. If your distro does not, or you
have need to compile bonding from source (e.g., configuring and
installing a mainline kernel from kernel.org), you'll need to perform
the following steps:
1.1 Configure and build the kernel with bonding
-----------------------------------------------
The current version of the bonding driver is available in the
drivers/net/bonding subdirectory of the most recent kernel source
(which is available on http://kernel.org). Most users "rolling their
own" will want to use the most recent kernel from kernel.org.
Configure kernel with "make menuconfig" (or "make xconfig" or
"make config"), then select "Bonding driver support" in the "Network
device support" section. It is recommended that you configure the
driver as module since it is currently the only way to pass parameters
to the driver or configure more than one bonding device.
Build and install the new kernel and modules.
1.2 Bonding Control Utility
---------------------------
It is recommended to configure bonding via iproute2 (netlink)
or sysfs, the old ifenslave control utility is obsolete.
2. Bonding Driver Options
=========================
Options for the bonding driver are supplied as parameters to the
bonding module at load time, or are specified via sysfs.
Module options may be given as command line arguments to the
insmod or modprobe command, but are usually specified in either the
``/etc/modprobe.d/*.conf`` configuration files, or in a distro-specific
configuration file (some of which are detailed in the next section).
Details on bonding support for sysfs is provided in the
"Configuring Bonding Manually via Sysfs" section, below.
The available bonding driver parameters are listed below. If a
parameter is not specified the default value is used. When initially
configuring a bond, it is recommended "tail -f /var/log/messages" be
run in a separate window to watch for bonding driver error messages.
It is critical that either the miimon or arp_interval and
arp_ip_target parameters be specified, otherwise serious network
degradation will occur during link failures. Very few devices do not
support at least miimon, so there is really no reason not to use it.
Options with textual values will accept either the text name
or, for backwards compatibility, the option value. E.g.,
"mode=802.3ad" and "mode=4" set the same mode.
The parameters are as follows:
Active slave와 802.3ad 선택 option
170-297`active_slave`: active-backup, balance-alb, balance-tlb에서 새 active slave를 지정합니다. 현재 enslave된 interface 이름 또는 빈 문자열을 받습니다. 선택 대상과 link가 up이어야 하며 빈 문자열은 현재 active를 지우고 자동 재선택합니다. Sysfs 전용이며 정상 조회값은 active interface 이름 또는 빈 문자열입니다.
`ad_actor_sys_prio`: 802.3ad actor system priority, 범위 1~65535, 기본 65535이며 sysfs에서 설정합니다. `actor_port_prio`: port priority 범위 1~65535, 기본 255이며 netlink에서 설정합니다.
`ad_actor_system`: LACPDU 교환에서 actor가 사용할 MAC address입니다. Multicast는 금지되고 all-zero MAC이면 bond MAC을 내부 사용합니다. Local-admin bit 설정을 권장하지만 강제하지 않으며 생략하면 master MAC을 actor system address로 사용합니다. 802.3ad·sysfs 전용입니다.
`ad_select`는 active aggregator 선택 방식을 정합니다. `stable(0)`은 최대 aggregate bandwidth를 고르고 현재 aggregator의 모든 slave가 down이거나 slave가 없을 때만 재선택하는 기본값입니다. `bandwidth(1)`는 최대 bandwidth를 고르되 slave 추가·삭제, link·association 변화, bond administrative up 때 재선택합니다. `count(2)`는 port 수가 가장 많은 aggregator를, `actor_port_prio(3)`는 active port의 actor priority 합이 가장 큰 aggregator를 고릅니다.
Bandwidth·count·actor_port_prio policy는 active aggregator 일부가 실패해도 남은 bandwidth, port 수 또는 priority 합이 가장 높은 aggregator로 failover합니다. 이 option은 bonding 3.4.0에 추가되었습니다.
`ad_user_port_key`: port key의 bit 0은 duplex, bit 1~5는 speed, bit 6~15는 user-defined입니다. 이 option은 상위 10 bit를 0~1023으로 설정하며 기본 0이고 802.3ad sysfs 전용입니다.
`all_slaves_active`: inactive port에서 받은 duplicate frame을 0이면 버리고 1이면 전달합니다. 대부분에는 drop이 적합하며 기본 0입니다.
active_slave
Specifies the new active slave for modes that support it
(active-backup, balance-alb and balance-tlb). Possible values
are the name of any currently enslaved interface, or an empty
string. If a name is given, the slave and its link must be up in order
to be selected as the new active slave. If an empty string is
specified, the current active slave is cleared, and a new active
slave is selected automatically.
Note that this is only available through the sysfs interface. No module
parameter by this name exists.
The normal value of this option is the name of the currently
active slave, or the empty string if there is no active slave or
the current mode does not use an active slave.
ad_actor_sys_prio
In an AD system, this specifies the system priority. The allowed range
is 1 - 65535. If the value is not specified, it takes 65535 as the
default value.
This parameter has effect only in 802.3ad mode and is available through
SysFs interface.
actor_port_prio
In an AD system, this specifies the port priority. The allowed range
is 1 - 65535. If the value is not specified, it takes 255 as the
default value.
This parameter has effect only in 802.3ad mode and is available through
netlink interface.
ad_actor_system
In an AD system, this specifies the mac-address for the actor in
protocol packet exchanges (LACPDUs). The value cannot be a multicast
address. If the all-zeroes MAC is specified, bonding will internally
use the MAC of the bond itself. It is preferred to have the
local-admin bit set for this mac but driver does not enforce it. If
the value is not given then system defaults to using the masters'
mac address as actors' system address.
This parameter has effect only in 802.3ad mode and is available through
SysFs interface.
ad_select
Specifies the 802.3ad aggregation selection logic to use. The
possible values and their effects are:
stable or 0
The active aggregator is chosen by largest aggregate
bandwidth.
Reselection of the active aggregator occurs only when all
slaves of the active aggregator are down or the active
aggregator has no slaves.
This is the default value.
bandwidth or 1
The active aggregator is chosen by largest aggregate
bandwidth. Reselection occurs if:
- A slave is added to or removed from the bond
- Any slave's link state changes
- Any slave's 802.3ad association state changes
- The bond's administrative state changes to up
count or 2
The active aggregator is chosen by the largest number of
ports (slaves). Reselection occurs as described under the
"bandwidth" setting, above.
actor_port_prio or 3
The active aggregator is chosen by the highest total sum of
actor port priorities across its active ports. Note this
priority is actor_port_prio, not per port prio, which is
used for primary reselect.
The bandwidth, count and actor_port_prio selection policies permit
failover of 802.3ad aggregations when partial failure of the active
aggregator occurs. This keeps the aggregator with the highest
availability (either in bandwidth, number of ports, or total value
of port priorities) active at all times.
This option was added in bonding version 3.4.0.
ad_user_port_key
In an AD system, the port-key has three parts as shown below -
===== ============
Bits Use
===== ============
00 Duplex
01-05 Speed
06-15 User-defined
===== ============
This defines the upper 10 bits of the port key. The values can be
from 0 - 1023. If not given, the system defaults to 0.
This parameter has effect only in 802.3ad mode and is available through
SysFs interface.
all_slaves_active
Specifies that duplicate frames (received on inactive ports) should be
dropped (0) or delivered (1).
Normally, bonding will drop duplicate frames (received on inactive
ports), which is desirable for most users. But there are some times
it is nice to allow duplicate frames to be delivered.
The default value is 0 (drop duplicate frames received on inactive
ports).
ARP·NS link monitoring option
298-475`arp_interval`: millisecond 단위 ARP monitoring 주기입니다. 각 slave가 최근 traffic을 송수신했는지 mode·상태별 기준으로 점검하고 `arp_ip_target`을 향한 probe로 정기 traffic을 만듭니다. 0은 비활성화이며 기본 0입니다. Mode 0·2에서 switch가 XOR로 reply를 한 link에 몰면 다른 member가 실패로 판정될 수 있으므로 고르게 분배하도록 구성해야 합니다. `miimon`과 함께 사용하지 않습니다.
`arp_ip_target`: ARP monitoring peer IPv4 address를 점 표기법으로 지정합니다. 여러 값은 comma로 구분하며 최소 1개, 최대 16개입니다. 기본값은 없음입니다. `ns_ip6_target`은 같은 역할의 IPv6 NS/NA target이며 IPv6 표기법, 최대 16개, 기본 없음입니다.
`arp_validate`는 ARP probe·reply 검증 또는 link 판단 시 non-ARP filtering을 정합니다. `none(0)`, active만 검증하는 `active(1)`, backup만 `backup(2)`, 모두 `all(3)`, filtering만 모든 slave에 하는 `filter(4)`, 전체 filtering+active 검증 `filter_active(5)`, 전체 filtering+backup 검증 `filter_backup(6)`을 제공합니다.
Validation은 들어온 ARP request·reply가 적절할 때만 slave를 up으로 봅니다. Active slave는 reply가 자신의 target에서 왔는지 확인하고, 보통 reply를 받지 못하는 backup은 active가 보낸 broadcast ARP request를 검사합니다. Switch 구성상 backup에 request가 전달되지 않으면 backup validation을 꺼야 합니다. 이는 다음 active 후보의 가능성을 판단할 뿐 실제 failover 성공을 보장하지 않습니다.
여러 bonding host가 공통 switch 너머 target에 동시에 probe할 때 switch-target link만 끊어지면 서로의 ARP가 표준 monitor를 속일 수 있습니다. Validation은 자기 instance와 관련된 ARP만 인정해 이를 해결합니다.
Filtering은 link availability 판단에 수신 ARP packet만 사용합니다. Non-ARP packet은 정상 전달하지만 link up 근거로 세지 않습니다. 제3자 broadcast가 많은 환경에서 표준 ARP monitor가 link를 살아 있다고 오판하는 문제를 줄입니다. `arp_validate`는 bonding 3.1.0에 추가되었습니다.
`arp_all_targets`는 validation이 켜진 active-backup slave를 up으로 보기 위해 target 몇 개가 reachable해야 하는지 정합니다. `any(0)`는 하나 이상, `all(1)`은 전부를 요구합니다.
`arp_missed_max`: interface를 down으로 표시하기 전에 연속 실패해야 할 monitor check 수입니다. Backup은 orderly failover를 위해 한 번 더, 즉 `arp_missed_max+1`회 실패해야 합니다. 범위 1~255, 기본 2입니다.
`coupled_control`: 802.3ad MUX state machine에서 Collecting과 Distributing을 분리할지 정합니다. IEEE 802.1AX-2008 5.4.15의 independent control을 기존 coupled control과 함께 구현합니다. 기본 1은 상태를 분리하지 않아 coupled control을 유지합니다.
arp_interval
Specifies the ARP link monitoring frequency in milliseconds.
The ARP monitor works by periodically checking the slave
devices to determine whether they have sent or received
traffic recently (the precise criteria depends upon the
bonding mode, and the state of the slave). Regular traffic is
generated via ARP probes issued for the addresses specified by
the arp_ip_target option.
This behavior can be modified by the arp_validate option,
below.
If ARP monitoring is used in an etherchannel compatible mode
(modes 0 and 2), the switch should be configured in a mode
that evenly distributes packets across all links. If the
switch is configured to distribute the packets in an XOR
fashion, all replies from the ARP targets will be received on
the same link which could cause the other team members to
fail. ARP monitoring should not be used in conjunction with
miimon. A value of 0 disables ARP monitoring. The default
value is 0.
arp_ip_target
Specifies the IP addresses to use as ARP monitoring peers when
arp_interval is > 0. These are the targets of the ARP request
sent to determine the health of the link to the targets.
Specify these values in ddd.ddd.ddd.ddd format. Multiple IP
addresses must be separated by a comma. At least one IP
address must be given for ARP monitoring to function. The
maximum number of targets that can be specified is 16. The
default value is no IP addresses.
ns_ip6_target
Specifies the IPv6 addresses to use as IPv6 monitoring peers when
arp_interval is > 0. These are the targets of the NS request
sent to determine the health of the link to the targets.
Specify these values in ffff:ffff::ffff:ffff format. Multiple IPv6
addresses must be separated by a comma. At least one IPv6
address must be given for NS/NA monitoring to function. The
maximum number of targets that can be specified is 16. The
default value is no IPv6 addresses.
arp_validate
Specifies whether or not ARP probes and replies should be
validated in any mode that supports arp monitoring, or whether
non-ARP traffic should be filtered (disregarded) for link
monitoring purposes.
Possible values are:
none or 0
No validation or filtering is performed.
active or 1
Validation is performed only for the active slave.
backup or 2
Validation is performed only for backup slaves.
all or 3
Validation is performed for all slaves.
filter or 4
Filtering is applied to all slaves. No validation is
performed.
filter_active or 5
Filtering is applied to all slaves, validation is performed
only for the active slave.
filter_backup or 6
Filtering is applied to all slaves, validation is performed
only for backup slaves.
Validation:
Enabling validation causes the ARP monitor to examine the incoming
ARP requests and replies, and only consider a slave to be up if it
is receiving the appropriate ARP traffic.
For an active slave, the validation checks ARP replies to confirm
that they were generated by an arp_ip_target. Since backup slaves
do not typically receive these replies, the validation performed
for backup slaves is on the broadcast ARP request sent out via the
active slave. It is possible that some switch or network
configurations may result in situations wherein the backup slaves
do not receive the ARP requests; in such a situation, validation
of backup slaves must be disabled.
The validation of ARP requests on backup slaves is mainly helping
bonding to decide which slaves are more likely to work in case of
the active slave failure, it doesn't really guarantee that the
backup slave will work if it's selected as the next active slave.
Validation is useful in network configurations in which multiple
bonding hosts are concurrently issuing ARPs to one or more targets
beyond a common switch. Should the link between the switch and
target fail (but not the switch itself), the probe traffic
generated by the multiple bonding instances will fool the standard
ARP monitor into considering the links as still up. Use of
validation can resolve this, as the ARP monitor will only consider
ARP requests and replies associated with its own instance of
bonding.
Filtering:
Enabling filtering causes the ARP monitor to only use incoming ARP
packets for link availability purposes. Arriving packets that are
not ARPs are delivered normally, but do not count when determining
if a slave is available.
Filtering operates by only considering the reception of ARP
packets (any ARP packet, regardless of source or destination) when
determining if a slave has received traffic for link availability
purposes.
Filtering is useful in network configurations in which significant
levels of third party broadcast traffic would fool the standard
ARP monitor into considering the links as still up. Use of
filtering can resolve this, as only ARP traffic is considered for
link availability purposes.
This option was added in bonding version 3.1.0.
arp_all_targets
Specifies the quantity of arp_ip_targets that must be reachable
in order for the ARP monitor to consider a slave as being up.
This option affects only active-backup mode for slaves with
arp_validation enabled.
Possible values are:
any or 0
consider the slave up only when any of the arp_ip_targets
is reachable
all or 1
consider the slave up only when all of the arp_ip_targets
are reachable
arp_missed_max
Specifies the number of arp_interval monitor checks that must
fail in order for an interface to be marked down by the ARP monitor.
In order to provide orderly failover semantics, backup interfaces
are permitted an extra monitor check (i.e., they must fail
arp_missed_max + 1 times before being marked down).
The default value is 2, and the allowable range is 1 - 255.
coupled_control
Specifies whether the LACP state machine's MUX in the 802.3ad mode
should have separate Collecting and Distributing states.
This is by implementing the independent control state machine per
IEEE 802.1AX-2008 5.4.15 in addition to the existing coupled control
state machine.
The default value is 1. This setting does not separate the Collecting
and Distributing states, maintaining the bond in coupled control.
Delay·MAC·LACP·carrier option
476-621`downdelay`: link failure 감지 후 slave를 disable하기까지 기다릴 millisecond입니다. `miimon` 전용이며 miimon의 배수여야 하고 아니면 아래 배수로 내림합니다. 기본 0입니다.
`fail_over_mac`은 active-backup에서 MAC 처리 policy를 정합니다. `none(0)`은 enslave 때 모든 slave를 같은 MAC으로 만드는 전통적 기본 방식입니다.
`active(1)`는 bond MAC을 항상 현재 active slave의 고유 MAC으로 만들어 failover 때 bond MAC이 바뀝니다. MAC 변경 불가 device나 자신의 source MAC broadcast를 거부하는 device에 유용하지만 network 전체가 gratuitous ARP로 갱신되어야 하므로 이를 잃으면 통신이 끊길 수 있습니다. Link up을 실제 송수신 가능 시점보다 일찍 보고하는 device는 적절한 `updelay`가 필요합니다.
`follow(2)`는 bond MAC을 보통 첫 slave에서 정하되 backup role의 다른 slave MAC은 바꾸지 않습니다. Failover 순간 새 active에 bond MAC을 설정하고 이전 active에는 새 active의 원래 MAC을 줍니다. 여러 port에 같은 MAC을 넣으면 혼란·성능 저하가 생기는 multiport device에 유용합니다. 첫 slave가 MAC을 바꿀 수 없으면 기본 policy가 active로 바뀝니다. Slave가 없을 때만 sysfs로 변경할 수 있습니다.
`lacp_active`: `off(0)`는 상대가 말할 때만 LACPDU를 보내고 `on(1)`은 주기 전송하며 기본 on입니다. `lacp_rate`: partner LACPDU 주기를 `slow(0)` 30초 또는 `fast(1)` 1초로 요청하며 기본 slow입니다.
`broadcast_neighbor`: 802.3ad의 모든 active slave로 ARP·ND packet을 broadcast할지 정하며 기본 off입니다.
`max_bonds`: driver instance가 만들 bond 수입니다. 3이면 bond0~bond2를 만들며 기본 1, 0이면 module만 load하고 device는 만들지 않습니다.
`miimon`: millisecond 단위 MII link check 주기입니다. 0은 비활성화, 100은 좋은 시작값이며 `arp_interval`이 없을 때 기본 100입니다.
`min_links`: 802.3ad bond carrier를 up으로 올리기 위한 최소 active link 수입니다. Cluster 같은 상위 service가 최소 bandwidth를 확보한 뒤 switchover하게 할 수 있습니다. 기본 0은 active aggregator가 있으면 link 수와 무관하게 carrier를 올리며 aggregator에는 최소 한 link가 필요하므로 0과 1의 효과가 같습니다.
downdelay
Specifies the time, in milliseconds, to wait before disabling
a slave after a link failure has been detected. This option
is only valid for the miimon link monitor. The downdelay
value should be a multiple of the miimon value; if not, it
will be rounded down to the nearest multiple. The default
value is 0.
fail_over_mac
Specifies whether active-backup mode should set all slaves to
the same MAC address at enslavement (the traditional
behavior), or, when enabled, perform special handling of the
bond's MAC address in accordance with the selected policy.
Possible values are:
none or 0
This setting disables fail_over_mac, and causes
bonding to set all slaves of an active-backup bond to
the same MAC address at enslavement time. This is the
default.
active or 1
The "active" fail_over_mac policy indicates that the
MAC address of the bond should always be the MAC
address of the currently active slave. The MAC
address of the slaves is not changed; instead, the MAC
address of the bond changes during a failover.
This policy is useful for devices that cannot ever
alter their MAC address, or for devices that refuse
incoming broadcasts with their own source MAC (which
interferes with the ARP monitor).
The down side of this policy is that every device on
the network must be updated via gratuitous ARP,
vs. just updating a switch or set of switches (which
often takes place for any traffic, not just ARP
traffic, if the switch snoops incoming traffic to
update its tables) for the traditional method. If the
gratuitous ARP is lost, communication may be
disrupted.
When this policy is used in conjunction with the mii
monitor, devices which assert link up prior to being
able to actually transmit and receive are particularly
susceptible to loss of the gratuitous ARP, and an
appropriate updelay setting may be required.
follow or 2
The "follow" fail_over_mac policy causes the MAC
address of the bond to be selected normally (normally
the MAC address of the first slave added to the bond).
However, the second and subsequent slaves are not set
to this MAC address while they are in a backup role; a
slave is programmed with the bond's MAC address at
failover time (and the formerly active slave receives
the newly active slave's MAC address).
This policy is useful for multiport devices that
either become confused or incur a performance penalty
when multiple ports are programmed with the same MAC
address.
The default policy is none, unless the first slave cannot
change its MAC address, in which case the active policy is
selected by default.
This option may be modified via sysfs only when no slaves are
present in the bond.
This option was added in bonding version 3.2.0. The "follow"
policy was added in bonding version 3.3.0.
lacp_active
Option specifying whether to send LACPDU frames periodically.
off or 0
LACPDU frames acts as "speak when spoken to".
on or 1
LACPDU frames are sent along the configured links
periodically. See lacp_rate for more details.
The default is on.
lacp_rate
Option specifying the rate in which we'll ask our link partner
to transmit LACPDU packets in 802.3ad mode. Possible values
are:
slow or 0
Request partner to transmit LACPDUs every 30 seconds
fast or 1
Request partner to transmit LACPDUs every 1 second
The default is slow.
broadcast_neighbor
Option specifying whether to broadcast ARP/ND packets to all
active slaves. This option has no effect in modes other than
802.3ad mode. The default is off (0).
max_bonds
Specifies the number of bonding devices to create for this
instance of the bonding driver. E.g., if max_bonds is 3, and
the bonding driver is not already loaded, then bond0, bond1
and bond2 will be created. The default value is 1. Specifying
a value of 0 will load bonding, but will not create any devices.
miimon
Specifies the MII link monitoring frequency in milliseconds.
This determines how often the link state of each slave is
inspected for link failures. A value of zero disables MII
link monitoring. A value of 100 is a good starting point.
The default value is 100 if arp_interval is not set.
min_links
Specifies the minimum number of links that must be active before
asserting carrier. It is similar to the Cisco EtherChannel min-links
feature. This allows setting the minimum number of member ports that
must be up (link-up state) before marking the bond device as up
(carrier on). This is useful for situations where higher level services
such as clustering want to ensure a minimum number of low bandwidth
links are active before switchover. This option only affect 802.3ad
mode.
The default value is 0. This will cause carrier to be asserted (for
802.3ad mode) whenever there is an active aggregator, regardless of the
number of available links in that aggregator. Note that, because an
aggregator cannot be active without at least one available link,
setting this option to 0 or to 1 has the exact same effect.
일곱 bonding mode
622-778`mode`의 기본은 `balance-rr(0)`이며 다음 policy를 제공합니다.
`balance-rr(0)`: 사용 가능한 첫 slave부터 마지막까지 packet을 순차 송신해 load balancing과 fault tolerance를 제공합니다.
`active-backup(1)`: 한 slave만 active이고 그 link가 실패해야 다른 slave가 active가 됩니다. Bond MAC은 switch 혼란을 피하도록 외부에서 한 port에만 보입니다. Failover 때 새 active로 bond master와 IP가 있는 각 VLAN interface의 gratuitous ARP를 보내며 VLAN ARP에는 tag를 붙입니다. Fault tolerance mode이고 `primary`가 동작에 영향을 줍니다.
`balance-xor(2)`: 선택한 transmit hash policy로 slave를 고릅니다. 기본은 source MAC XOR destination MAC XOR packet type ID를 slave 수로 나눈 값입니다. Load balancing과 fault tolerance를 제공합니다.
`broadcast(3)`: 모든 slave로 모든 packet을 보내 fault tolerance를 제공합니다.
`802.3ad(4)`: IEEE dynamic link aggregation입니다. 같은 speed와 duplex의 link를 aggregator로 묶고 active aggregator의 모든 slave를 사용합니다. 송신 slave는 `xmit_hash_policy`로 선택합니다. 일부 policy는 packet 순서를 요구하는 표준 43.2.4를 완전히 지키지 않을 수 있습니다. Base driver의 ethtool speed·duplex 조회와 switch의 802.3ad 지원·구성이 필요합니다.
`balance-tlb(5)`: 특별한 switch 지원 없이 adaptive transmit load balancing을 합니다. `tlb_dynamic_lb=1`은 각 slave speed 대비 현재 load로 송신을 분배하고 0은 hash distribution만 씁니다. 수신은 현재 slave 한 개가 담당하고 실패하면 다른 slave가 그 MAC을 이어받습니다. Base driver에서 slave speed를 조회하는 ethtool 지원이 필요합니다.
`balance-alb(6)`: balance-tlb에 IPv4 receive load balancing을 더하며 특별한 switch 지원이 필요 없습니다. Bonding driver는 local ARP reply의 source hardware address를 slave별 고유 MAC으로 바꿔 peer별 수신 traffic을 분배합니다. Local system이 ARP request를 보낼 때 peer IP를 저장하고 reply가 오면 peer에 특정 slave MAC을 알려 줍니다.
Broadcast ARP request가 bond MAC을 다시 학습시켜 수신이 current slave에 몰리는 문제는 peer마다 할당한 MAC의 ARP reply update를 보내 재분배합니다. Slave 추가·재활성화 때도 가장 빠른 slave group에 round-robin으로 재분배합니다. Switch forwarding delay 이상으로 `updelay`를 설정해야 ARP update가 막히지 않습니다.
Balance-alb에는 ethtool speed 조회와, device가 열린 상태에서 hardware address를 바꾸는 base driver 지원이 필요합니다. 항상 한 slave는 bond MAC을 사용하고 나머지는 고유 MAC을 가지며 current active가 실패하면 새 active와 MAC을 교환합니다.
mode
Specifies one of the bonding policies. The default is
balance-rr (round robin). Possible values are:
balance-rr or 0
Round-robin policy: Transmit packets in sequential
order from the first available slave through the
last. This mode provides load balancing and fault
tolerance.
active-backup or 1
Active-backup policy: Only one slave in the bond is
active. A different slave becomes active if, and only
if, the active slave fails. The bond's MAC address is
externally visible on only one port (network adapter)
to avoid confusing the switch.
In bonding version 2.6.2 or later, when a failover
occurs in active-backup mode, bonding will issue one
or more gratuitous ARPs on the newly active slave.
One gratuitous ARP is issued for the bonding master
interface and each VLAN interfaces configured above
it, provided that the interface has at least one IP
address configured. Gratuitous ARPs issued for VLAN
interfaces are tagged with the appropriate VLAN id.
This mode provides fault tolerance. The primary
option, documented below, affects the behavior of this
mode.
balance-xor or 2
XOR policy: Transmit based on the selected transmit
hash policy. The default policy is a simple [(source
MAC address XOR'd with destination MAC address XOR
packet type ID) modulo slave count]. Alternate transmit
policies may be selected via the xmit_hash_policy option,
described below.
This mode provides load balancing and fault tolerance.
broadcast or 3
Broadcast policy: transmits everything on all slave
interfaces. This mode provides fault tolerance.
802.3ad or 4
IEEE 802.3ad Dynamic link aggregation. Creates
aggregation groups that share the same speed and
duplex settings. Utilizes all slaves in the active
aggregator according to the 802.3ad specification.
Slave selection for outgoing traffic is done according
to the transmit hash policy, which may be changed from
the default simple XOR policy via the xmit_hash_policy
option, documented below. Note that not all transmit
policies may be 802.3ad compliant, particularly in
regards to the packet mis-ordering requirements of
section 43.2.4 of the 802.3ad standard. Differing
peer implementations will have varying tolerances for
noncompliance.
Prerequisites:
1. Ethtool support in the base drivers for retrieving
the speed and duplex of each slave.
2. A switch that supports IEEE 802.3ad Dynamic link
aggregation.
Most switches will require some type of configuration
to enable 802.3ad mode.
balance-tlb or 5
Adaptive transmit load balancing: channel bonding that
does not require any special switch support.
In tlb_dynamic_lb=1 mode; the outgoing traffic is
distributed according to the current load (computed
relative to the speed) on each slave.
In tlb_dynamic_lb=0 mode; the load balancing based on
current load is disabled and the load is distributed
only using the hash distribution.
Incoming traffic is received by the current slave.
If the receiving slave fails, another slave takes over
the MAC address of the failed receiving slave.
Prerequisite:
Ethtool support in the base drivers for retrieving the
speed of each slave.
balance-alb or 6
Adaptive load balancing: includes balance-tlb plus
receive load balancing (rlb) for IPV4 traffic, and
does not require any special switch support. The
receive load balancing is achieved by ARP negotiation.
The bonding driver intercepts the ARP Replies sent by
the local system on their way out and overwrites the
source hardware address with the unique hardware
address of one of the slaves in the bond such that
different peers use different hardware addresses for
the server.
Receive traffic from connections created by the server
is also balanced. When the local system sends an ARP
Request the bonding driver copies and saves the peer's
IP information from the ARP packet. When the ARP
Reply arrives from the peer, its hardware address is
retrieved and the bonding driver initiates an ARP
reply to this peer assigning it to one of the slaves
in the bond. A problematic outcome of using ARP
negotiation for balancing is that each time that an
ARP request is broadcast it uses the hardware address
of the bond. Hence, peers learn the hardware address
of the bond and the balancing of receive traffic
collapses to the current slave. This is handled by
sending updates (ARP Replies) to all the peers with
their individually assigned hardware address such that
the traffic is redistributed. Receive traffic is also
redistributed when a new slave is added to the bond
and when an inactive slave is re-activated. The
receive load is distributed sequentially (round robin)
among the group of highest speed slaves in the bond.
When a link is reconnected or a new slave joins the
bond the receive traffic is redistributed among all
active slaves in the bond by initiating ARP Replies
with the selected MAC address to each of the
clients. The updelay parameter (detailed below) must
be set to a value equal or greater than the switch's
forwarding delay so that the ARP Replies sent to the
peers will not be blocked by the switch.
Prerequisites:
1. Ethtool support in the base drivers for retrieving
the speed of each slave.
2. Base driver support for setting the hardware
address of a device while it is open. This is
required so that there will always be one slave in the
team using the bond hardware address (the
curr_active_slave) while having a unique hardware
address for each slave in the bond. If the
curr_active_slave fails its hardware address is
swapped with the new curr_active_slave that was
chosen.
Peer 통지·primary·TLB option
779-922`num_grat_arp`, `num_unsol_na`: failover 뒤 보낼 gratuitous ARP와 unsolicited IPv6 Neighbor Advertisement 반복 횟수입니다. 새 slave link가 up이면 bond와 각 VLAN sub-device에서 즉시 보내고 1보다 크면 `peer_notif_delay` 간격으로 반복합니다. 범위 0~255, 기본 1이며 active-backup 또는 `broadcast_neighbor`가 켜진 802.3ad에 적용됩니다. Linux 3.0·bonding 3.7.1부터 IPv4·IPv6 code가 생성해 두 반복 횟수를 독립 설정할 수 없습니다.
`packets_per_slave`: balance-rr에서 다음 slave로 넘어가기 전에 보낼 packet 수입니다. 0이면 random slave를 고릅니다. 범위 0~65535, 기본 1입니다.
`peer_notif_delay`: failover 뒤 peer notification 사이의 millisecond 간격입니다. `miimon`의 배수여야 하며 범위 0~300000, 기본 0은 miimon과 같은 값입니다.
`prio`: slave priority이며 숫자가 클수록 우선입니다. Primary가 최고 priority이고 `primary_reselect` 규칙을 따릅니다. Netlink 전용이며 active-backup, balance-tlb, balance-alb에서만 유효합니다. Signed 32-bit 범위, 기본 0입니다.
`primary`: 항상 선호할 slave 이름입니다. 사용 가능하면 active를 유지하고 offline일 때만 대체 slave를 씁니다. Slave별 처리량이 다를 때 유용하며 active-backup, balance-tlb, balance-alb에서만 유효합니다.
`primary_reselect`: primary가 복구될 때 다시 active로 만들 조건입니다. `always(0)`는 복구 즉시 되돌리는 기본값, `better(1)`는 현재 active보다 speed·duplex가 좋을 때, `failure(2)`는 현재 active가 실패했을 때만 되돌립니다. Active slave가 하나도 없으면 먼저 복구된 slave를, 최초 enslave 때는 primary를 선택합니다. Sysfs 변경은 새 policy에 따른 즉시 재선택을 유발할 수 있습니다.
`tlb_dynamic_lb`: TLB·ALB에서 interval별 load에 따라 active flow를 slave 사이에 재배치할지 정합니다. 기본 1은 좋은 balancing 대신 packet reorder 가능성이 있고 0은 재배치를 끄고 `xmit_hash_policy` 분배만 사용합니다. Bond가 down일 때만 sysfs로 바꿀 수 있습니다.
`updelay`: link recovery 감지 후 slave enable까지 기다릴 millisecond이며 miimon의 배수로 내림됩니다. 기본 0입니다.
`use_carrier`: 과거 MII·ETHTOOL ioctl과 `netif_carrier_ok()` 중 선택하던 obsolete option입니다. 현재 모든 check가 `netif_carrier_ok()`를 쓰며 호환용으로 읽고 쓸 수 있는 유일한 값은 1입니다.
num_grat_arp,
num_unsol_na
Specify the number of peer notifications (gratuitous ARPs and
unsolicited IPv6 Neighbor Advertisements) to be issued after a
failover event. As soon as the link is up on the new slave
(possibly immediately) a peer notification is sent on the
bonding device and each VLAN sub-device. This is repeated at
the rate specified by peer_notif_delay if the number is
greater than 1.
The valid range is 0 - 255; the default value is 1. These options
affect the active-backup or 802.3ad (broadcast_neighbor enabled) mode.
These options were added for bonding versions 3.3.0 and 3.4.0
respectively.
From Linux 3.0 and bonding version 3.7.1, these notifications
are generated by the ipv4 and ipv6 code and the numbers of
repetitions cannot be set independently.
packets_per_slave
Specify the number of packets to transmit through a slave before
moving to the next one. When set to 0 then a slave is chosen at
random.
The valid range is 0 - 65535; the default value is 1. This option
has effect only in balance-rr mode.
peer_notif_delay
Specify the delay, in milliseconds, between each peer
notification (gratuitous ARP and unsolicited IPv6 Neighbor
Advertisement) when they are issued after a failover event.
This delay should be a multiple of the MII link monitor interval
(miimon).
The valid range is 0 - 300000. The default value is 0, which means
to match the value of the MII link monitor interval.
prio
Slave priority. A higher number means higher priority.
The primary slave has the highest priority. This option also
follows the primary_reselect rules.
This option could only be configured via netlink, and is only valid
for active-backup(1), balance-tlb (5) and balance-alb (6) mode.
The valid value range is a signed 32 bit integer.
The default value is 0.
primary
A string (eth0, eth2, etc) specifying which slave is the
primary device. The specified device will always be the
active slave while it is available. Only when the primary is
off-line will alternate devices be used. This is useful when
one slave is preferred over another, e.g., when one slave has
higher throughput than another.
The primary option is only valid for active-backup(1),
balance-tlb (5) and balance-alb (6) mode.
primary_reselect
Specifies the reselection policy for the primary slave. This
affects how the primary slave is chosen to become the active slave
when failure of the active slave or recovery of the primary slave
occurs. This option is designed to prevent flip-flopping between
the primary slave and other slaves. Possible values are:
always or 0 (default)
The primary slave becomes the active slave whenever it
comes back up.
better or 1
The primary slave becomes the active slave when it comes
back up, if the speed and duplex of the primary slave is
better than the speed and duplex of the current active
slave.
failure or 2
The primary slave becomes the active slave only if the
current active slave fails and the primary slave is up.
The primary_reselect setting is ignored in two cases:
If no slaves are active, the first slave to recover is
made the active slave.
When initially enslaved, the primary slave is always made
the active slave.
Changing the primary_reselect policy via sysfs will cause an
immediate selection of the best active slave according to the new
policy. This may or may not result in a change of the active
slave, depending upon the circumstances.
This option was added for bonding version 3.6.0.
tlb_dynamic_lb
Specifies if dynamic shuffling of flows is enabled in tlb
or alb mode. The value has no effect on any other modes.
The default behavior of tlb mode is to shuffle active flows across
slaves based on the load in that interval. This gives nice lb
characteristics but can cause packet reordering. If re-ordering is
a concern use this variable to disable flow shuffling and rely on
load balancing provided solely by the hash distribution.
xmit-hash-policy can be used to select the appropriate hashing for
the setup.
The sysfs entry can be used to change the setting per bond device
and the initial value is derived from the module parameter. The
sysfs entry is allowed to be changed only if the bond device is
down.
The default value is "1" that enables flow shuffling while value "0"
disables it. This option was added in bonding driver 3.7.1
updelay
Specifies the time, in milliseconds, to wait before enabling a
slave after a link recovery has been detected. This option is
only valid for the miimon link monitor. The updelay value
should be a multiple of the miimon value; if not, it will be
rounded down to the nearest multiple. The default value is 0.
use_carrier
Obsolete option that previously selected between MII /
ETHTOOL ioctls and netif_carrier_ok() to determine link
state.
All link state checks are now done with netif_carrier_ok().
For backwards compatibility, this option's value may be inspected
or set. The only valid setting is 1.
Transmit hash·IGMP·learning packet
923-1069`xmit_hash_policy`는 balance-xor, 802.3ad, TLB의 송신 slave 선택 hash를 정합니다.
`layer2`: `source MAC[5] XOR destination MAC[5] XOR packet type ID`를 slave 수로 나눕니다. 특정 peer의 모든 traffic이 같은 slave를 사용하며 802.3ad compliant인 기본값입니다.
`layer2+3`: L2 hash에 source·destination IP를 XOR하고 16·8 bit shift-fold한 뒤 slave 수로 나눕니다. IPv6 address는 먼저 `ipv6_addr_hash`를 적용합니다. Gateway를 통해 많은 destination에 가는 환경에서 layer2보다 고르게 분배하며 802.3ad compliant입니다.
`layer3+4`: unfragmented TCP·UDP의 source·destination port와 IP를 fold합니다. 한 connection은 한 slave에 남지만 같은 peer로 가는 여러 connection은 여러 slave를 쓸 수 있습니다. Fragment packet은 port를 빼므로 fragmented·unfragmented가 섞인 한 conversation은 두 interface에 걸쳐 순서가 뒤바뀔 수 있어 완전한 802.3ad compliance는 아닙니다.
`encap2+3`, `encap3+4`: 각각 layer2+3, layer3+4 공식을 사용하되 `skb_flow_dissect`로 header를 얻어 tunnel이면 inner header를 hash할 수 있습니다. Encapsulated flow 기준 분배로 tunnel 성능을 개선합니다.
`vlan+srcmac`: `(VLAN ID) XOR (source MAC vendor) XOR (source MAC device)`의 단순 hash로 VLAN별 load balancing과 failover를 제공합니다. VLAN별 VM이 공유하는 bond에 LACP switch 없이 LACP 비슷한 동작을 제공하려는 용도입니다. Native XDP는 지원하지 않습니다.
`resend_igmp`: failover 후 보낼 IGMP membership report 수입니다. 하나는 즉시, 나머지는 200ms 간격입니다. 범위 0~255, 기본 1이고 0은 보내지 않습니다. balance-rr, active-backup, balance-tlb, balance-alb에서 IGMP 수신을 새 slave로 옮기기 위해 유용합니다.
`lp_interval`: balance-tlb·balance-alb에서 각 slave의 peer switch로 learning packet을 보내는 간격(초)입니다. 범위 1~0x7fffffff, 기본 1입니다.
xmit_hash_policy
Selects the transmit hash policy to use for slave selection in
balance-xor, 802.3ad, and tlb modes. Possible values are:
layer2
Uses XOR of hardware MAC addresses and packet type ID
field to generate the hash. The formula is
hash = source MAC[5] XOR destination MAC[5] XOR packet type ID
slave number = hash modulo slave count
This algorithm will place all traffic to a particular
network peer on the same slave.
This algorithm is 802.3ad compliant.
layer2+3
This policy uses a combination of layer2 and layer3
protocol information to generate the hash.
Uses XOR of hardware MAC addresses and IP addresses to
generate the hash. The formula is
hash = source MAC[5] XOR destination MAC[5] XOR packet type ID
hash = hash XOR source IP XOR destination IP
hash = hash XOR (hash RSHIFT 16)
hash = hash XOR (hash RSHIFT 8)
And then hash is reduced modulo slave count.
If the protocol is IPv6 then the source and destination
addresses are first hashed using ipv6_addr_hash.
This algorithm will place all traffic to a particular
network peer on the same slave. For non-IP traffic,
the formula is the same as for the layer2 transmit
hash policy.
This policy is intended to provide a more balanced
distribution of traffic than layer2 alone, especially
in environments where a layer3 gateway device is
required to reach most destinations.
This algorithm is 802.3ad compliant.
layer3+4
This policy uses upper layer protocol information,
when available, to generate the hash. This allows for
traffic to a particular network peer to span multiple
slaves, although a single connection will not span
multiple slaves.
The formula for unfragmented TCP and UDP packets is
hash = source port, destination port (as in the header)
hash = hash XOR source IP XOR destination IP
hash = hash XOR (hash RSHIFT 16)
hash = hash XOR (hash RSHIFT 8)
hash = hash RSHIFT 1
And then hash is reduced modulo slave count.
If the protocol is IPv6 then the source and destination
addresses are first hashed using ipv6_addr_hash.
For fragmented TCP or UDP packets and all other IPv4 and
IPv6 protocol traffic, the source and destination port
information is omitted. For non-IP traffic, the
formula is the same as for the layer2 transmit hash
policy.
This algorithm is not fully 802.3ad compliant. A
single TCP or UDP conversation containing both
fragmented and unfragmented packets will see packets
striped across two interfaces. This may result in out
of order delivery. Most traffic types will not meet
this criteria, as TCP rarely fragments traffic, and
most UDP traffic is not involved in extended
conversations. Other implementations of 802.3ad may
or may not tolerate this noncompliance.
encap2+3
This policy uses the same formula as layer2+3 but it
relies on skb_flow_dissect to obtain the header fields
which might result in the use of inner headers if an
encapsulation protocol is used. For example this will
improve the performance for tunnel users because the
packets will be distributed according to the encapsulated
flows.
encap3+4
This policy uses the same formula as layer3+4 but it
relies on skb_flow_dissect to obtain the header fields
which might result in the use of inner headers if an
encapsulation protocol is used. For example this will
improve the performance for tunnel users because the
packets will be distributed according to the encapsulated
flows.
vlan+srcmac
This policy uses a very rudimentary vlan ID and source mac
hash to load-balance traffic per-vlan, with failover
should one leg fail. The intended use case is for a bond
shared by multiple virtual machines, all configured to
use their own vlan, to give lacp-like functionality
without requiring lacp-capable switching hardware.
The formula for the hash is simply
hash = (vlan ID) XOR (source MAC vendor) XOR (source MAC dev)
The default value is layer2. This option was added in bonding
version 2.6.3. In earlier versions of bonding, this parameter
does not exist, and the layer2 policy is the only policy. The
layer2+3 value was added for bonding version 3.2.2.
resend_igmp
Specifies the number of IGMP membership reports to be issued after
a failover event. One membership report is issued immediately after
the failover, subsequent packets are sent in each 200ms interval.
The valid range is 0 - 255; the default value is 1. A value of 0
prevents the IGMP membership report from being issued in response
to the failover event.
This option is useful for bonding modes balance-rr (0), active-backup
(1), balance-tlb (5) and balance-alb (6), in which a failover can
switch the IGMP traffic from one slave to another. Therefore a fresh
IGMP report must be issued to cause the switch to forward the incoming
IGMP traffic over the newly selected slave.
This option was added for bonding version 3.7.0.
lp_interval
Specifies the number of seconds between instances where the bonding
driver sends learning packets to each slaves peer switch.
The valid range is 1 - 0x7fffffff; the default value is 1. This Option
has effect only in balance-tlb and balance-alb modes.
구성 체계 판별
1070-11103. Bonding device 구성
Distribution network initialization script, iproute2 또는 sysfs를 사용할 수 있습니다. Distro script package는 주로 initscripts, sysconfig, interfaces이며 최신 version은 bonding을 지원하지만 옛 version은 그렇지 않습니다.
`/etc/network/interfaces`가 있으면 interfaces 방식을 사용합니다. 없으면 `rpm -qf /sbin/ifup` 결과가 `initscripts` 또는 `sysconfig`로 시작하며 package를 알 수 있습니다. `grep ifenslave /sbin/ifup`가 match를 반환하면 해당 script에 bonding support가 있습니다.
3. Configuring Bonding Devices
==============================
You can configure bonding using either your distro's network
initialization scripts, or manually using either iproute2 or the
sysfs interface. Distros generally use one of three packages for the
network initialization scripts: initscripts, sysconfig or interfaces.
Recent versions of these packages have support for bonding, while older
versions do not.
We will first describe the options for configuring bonding for
distros using versions of initscripts, sysconfig and interfaces with full
or partial support for bonding, then provide information on enabling
bonding without support from the network initialization scripts (i.e.,
older versions of initscripts or sysconfig).
If you're unsure whether your distro uses sysconfig,
initscripts or interfaces, or don't know if it's new enough, have no fear.
Determining this is fairly straightforward.
First, look for a file called interfaces in /etc/network directory.
If this file is present in your system, then your system use interfaces. See
Configuration with Interfaces Support.
Else, issue the command::
$ rpm -qf /sbin/ifup
It will respond with a line of text starting with either
"initscripts" or "sysconfig," followed by some numbers. This is the
package that provides your network initialization scripts.
Next, to determine if your installation supports bonding,
issue the command::
$ grep ifenslave /sbin/ifup
If this returns any matches, then your initscripts or
sysconfig has support for bonding.
3.1 Configuration with Sysconfig SupportSysconfig 구성
1111-12673.1 Sysconfig 지원 구성은 예를 들어 SLES 9에 적용됩니다. YaST front end는 bonding device를 다루지 못하므로 file을 직접 관리합니다.
먼저 각 slave의 `ifcfg-id-xx:xx:xx:xx:xx:xx`를 만듭니다. SLES 9에서는 YaST2 sysconfig에서 임시로 DHCP 설정해 file을 생성할 수 있습니다. 각 file은 permanent MAC을 이름에 넣습니다. `BOOTPROTO='none'`, `STARTMODE='off'`로 바꾸고 `UNIQUE`, `_nm_name`은 유지하며 나머지 line은 제거합니다.
Bond file은 `ifcfg-bondX`이며 bond0, bond1처럼 증가합니다. `BOOTPROTO`, broadcast·IP·netmask·network, `STARTMODE`, `BONDING_MASTER='yes'`, `BONDING_MODULE_OPTS`, `BONDING_SLAVEn`을 설정합니다. Sample network 값을 실제 값으로 바꿉니다.
`STARTMODE=onboot`는 boot 시 시작, `manual`은 수동 ifup 때만, `hotplug`는 bonding에 부적합, `off`·`ignore`는 구성을 무시합니다.
`BONDING_MODULE_OPTS`에는 mode와 link monitoring 등을 넣되 `max_bonds`를 넣으면 여러 bond에서 sysconfig를 혼란시키므로 금지합니다. 각 slave는 interface 이름 또는 `bus-pci-...` physical specifier로 지정합니다. Interface 이름은 쉽게 찾지만 boot 때 순서가 바뀔 수 있고 bus 위치는 card slot을 옮기지 않는 한 안정적입니다.
File을 수정한 뒤 `/etc/init.d/network restart`를 실행합니다. Shutdown script가 bonding module도 제거하므로 parameter 변경 시 수동 `rmmod`가 필요 없습니다. YaST는 bonding interface를 표시하지 않아 변경도 file을 직접 편집해야 합니다. 일반 ifcfg option은 `/etc/sysconfig/network/ifcfg.template`에 있지만 `BONDING_*`는 설명하지 않습니다.
3.1.1 Sysconfig의 DHCP는 당시 slave를 추가하기 전에 DHCP를 시도해 request가 network로 나가지 않으므로 bond에서 작동하지 않았습니다.
3.1.2 여러 bond는 각 instance의 `ifcfg-bondX`를 만들면 됩니다. `max_bonds`는 지정하지 않습니다. 같은 parameter의 bond도 file을 각각 만들며 module option이 bond file에 있으므로 `/etc/modules.d/*.conf`에 추가할 필요가 없습니다.
----------------------------------------
This section applies to distros using a version of sysconfig
with bonding support, for example, SuSE Linux Enterprise Server 9.
SuSE SLES 9's networking configuration system does support
bonding, however, at this writing, the YaST system configuration
front end does not provide any means to work with bonding devices.
Bonding devices can be managed by hand, however, as follows.
First, if they have not already been configured, configure the
slave devices. On SLES 9, this is most easily done by running the
yast2 sysconfig configuration utility. The goal is for to create an
ifcfg-id file for each slave device. The simplest way to accomplish
this is to configure the devices for DHCP (this is only to get the
file ifcfg-id file created; see below for some issues with DHCP). The
name of the configuration file for each device will be of the form::
ifcfg-id-xx:xx:xx:xx:xx:xx
Where the "xx" portion will be replaced with the digits from
the device's permanent MAC address.
Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
created, it is necessary to edit the configuration files for the slave
devices (the MAC addresses correspond to those of the slave devices).
Before editing, the file will contain multiple lines, and will look
something like this::
BOOTPROTO='dhcp'
STARTMODE='on'
USERCTL='no'
UNIQUE='XNzu.WeZGOGF+4wE'
_nm_name='bus-pci-0001:61:01.0'
Change the BOOTPROTO and STARTMODE lines to the following::
BOOTPROTO='none'
STARTMODE='off'
Do not alter the UNIQUE or _nm_name lines. Remove any other
lines (USERCTL, etc).
Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified,
it's time to create the configuration file for the bonding device
itself. This file is named ifcfg-bondX, where X is the number of the
bonding device to create, starting at 0. The first such file is
ifcfg-bond0, the second is ifcfg-bond1, and so on. The sysconfig
network configuration system will correctly start multiple instances
of bonding.
The contents of the ifcfg-bondX file is as follows::
BOOTPROTO="static"
BROADCAST="10.0.2.255"
IPADDR="10.0.2.10"
NETMASK="255.255.0.0"
NETWORK="10.0.2.0"
REMOTE_IPADDR=""
STARTMODE="onboot"
BONDING_MASTER="yes"
BONDING_MODULE_OPTS="mode=active-backup miimon=100"
BONDING_SLAVE0="eth0"
BONDING_SLAVE1="bus-pci-0000:06:08.1"
Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
values with the appropriate values for your network.
The STARTMODE specifies when the device is brought online.
The possible values are:
======== ======================================================
onboot The device is started at boot time. If you're not
sure, this is probably what you want.
manual The device is started only when ifup is called
manually. Bonding devices may be configured this
way if you do not wish them to start automatically
at boot for some reason.
hotplug The device is started by a hotplug event. This is not
a valid choice for a bonding device.
off or The device configuration is ignored.
ignore
======== ======================================================
The line BONDING_MASTER='yes' indicates that the device is a
bonding master device. The only useful value is "yes."
The contents of BONDING_MODULE_OPTS are supplied to the
instance of the bonding module for this device. Specify the options
for the bonding mode, link monitoring, and so on here. Do not include
the max_bonds bonding parameter; this will confuse the configuration
system if you have multiple bonding devices.
Finally, supply one BONDING_SLAVEn="slave device" for each
slave. where "n" is an increasing value, one for each slave. The
"slave device" is either an interface name, e.g., "eth0", or a device
specifier for the network device. The interface name is easier to
find, but the ethN names are subject to change at boot time if, e.g.,
a device early in the sequence has failed. The device specifiers
(bus-pci-0000:06:08.1 in the example above) specify the physical
network device, and will not change unless the device's bus location
changes (for example, it is moved from one PCI slot to another). The
example above uses one of each type for demonstration purposes; most
configurations will choose one or the other for all slave devices.
When all configuration files have been modified or created,
networking must be restarted for the configuration changes to take
effect. This can be accomplished via the following::
# /etc/init.d/network restart
Note that the network control script (/sbin/ifdown) will
remove the bonding module as part of the network shutdown processing,
so it is not necessary to remove the module by hand if, e.g., the
module parameters have changed.
Also, at this writing, YaST/YaST2 will not manage bonding
devices (they do not show bonding interfaces on its list of network
devices). It is necessary to edit the configuration file by hand to
change the bonding configuration.
Additional general options and details of the ifcfg file
format can be found in an example ifcfg template file::
/etc/sysconfig/network/ifcfg.template
Note that the template does not document the various ``BONDING_*``
settings described above, but does describe many of the other options.
3.1.1 Using DHCP with Sysconfig
-------------------------------
Under sysconfig, configuring a device with BOOTPROTO='dhcp'
will cause it to query DHCP for its IP address information. At this
writing, this does not function for bonding devices; the scripts
attempt to obtain the device address from DHCP prior to adding any of
the slave devices. Without active slaves, the DHCP requests are not
sent to the network.
3.1.2 Configuring Multiple Bonds with Sysconfig
-----------------------------------------------
The sysconfig network initialization system is capable of
handling multiple bonding devices. All that is necessary is for each
bonding instance to have an appropriately configured ifcfg-bondX file
(as described above). Do not specify the "max_bonds" parameter to any
instance of bonding, as this will confuse sysconfig. If you require
multiple bonding devices with identical parameters, create multiple
ifcfg-bondX files.
Because the sysconfig scripts supply the bonding module
options in the ifcfg-bondX file, it is not necessary to add them to
the system ``/etc/modules.d/*.conf`` configuration files.
Initscripts 구성
1268-13863.2 Initscripts 지원 구성은 RHEL 3 이상, Fedora 등의 최신 package에 적용됩니다. IP가 없는 adapter driver를 자동 load하지 않는 제약 때문에 bond member가 될 모든 physical adapter의 `/etc/sysconfig/network-scripts/ifcfg-ethX`를 직접 만듭니다.
Slave file에는 `DEVICE=eth0`, `USERCTL=no`, `ONBOOT=yes`, `MASTER=bond0`, `SLAVE=yes`, `BOOTPROTO=none`을 넣습니다. File 이름과 DEVICE가 일치해야 하고 MASTER는 대상 bond 번호와 맞춥니다.
`ifcfg-bondX`에는 DEVICE, IPADDR, NETMASK, NETWORK, BROADCAST, ONBOOT, BOOTPROTO, USERCTL을 넣고 network 값은 환경에 맞게 바꿉니다.
Fedora 7·RHEL 5 이후에는 bond file의 `BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=..."`가 권장됩니다. 아주 오래된 initscripts는 여러 ARP target 각각에 `+`를 붙입니다. `BONDING_OPTS`를 쓰면 `/etc/modprobe.d/*.conf` 수정이 필요 없습니다.
`BONDING_OPTS` 미지원 package는 modprobe config에 `alias bond0 bonding`, `options bond0 mode=... miimon=100`을 넣습니다. 이후 root로 network restart를 실행합니다.
3.2.1 DHCP는 최신 initscripts에서 `BOOTPROTO=dhcp`와 대소문자를 지키는 `TYPE=Bonding`을 사용합니다.
3.2.2 Fedora 7·RHEL 5의 여러 bond는 각 `ifcfg-bondX`에 `BONDING_OPTS`를 넣습니다. Kernel sysfs와 bonding 3.0.0 이상이 필요하며 다른 환경은 수동 multiple bond 절을 따릅니다.
3.2 Configuration with Initscripts Support
------------------------------------------
This section applies to distros using a recent version of
initscripts with bonding support, for example, Red Hat Enterprise Linux
version 3 or later, Fedora, etc. On these systems, the network
initialization scripts have knowledge of bonding, and can be configured to
control bonding devices. Note that older versions of the initscripts
package have lower levels of support for bonding; this will be noted where
applicable.
These distros will not automatically load the network adapter
driver unless the ethX device is configured with an IP address.
Because of this constraint, users must manually configure a
network-script file for all physical adapters that will be members of
a bondX link. Network script files are located in the directory:
/etc/sysconfig/network-scripts
The file name must be prefixed with "ifcfg-eth" and suffixed
with the adapter's physical adapter number. For example, the script
for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0.
Place the following text in the file::
DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
The DEVICE= line will be different for every ethX device and
must correspond with the name of the file, i.e., ifcfg-eth1 must have
a device line of DEVICE=eth1. The setting of the MASTER= line will
also depend on the final bonding interface name chosen for your bond.
As with other network devices, these typically start at 0, and go up
one for each device, i.e., the first bonding instance is bond0, the
second is bond1, and so on.
Next, create a bond network script. The file name for this
script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is
the number of the bond. For bond0 the file is named "ifcfg-bond0",
for bond1 it is named "ifcfg-bond1", and so on. Within that file,
place the following text::
DEVICE=bond0
IPADDR=192.168.1.1
NETMASK=255.255.255.0
NETWORK=192.168.1.0
BROADCAST=192.168.1.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
Be sure to change the networking specific lines (IPADDR,
NETMASK, NETWORK and BROADCAST) to match your network configuration.
For later versions of initscripts, such as that found with Fedora
7 (or later) and Red Hat Enterprise Linux version 5 (or later), it is possible,
and, indeed, preferable, to specify the bonding options in the ifcfg-bond0
file, e.g. a line of the format::
BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=192.168.1.254"
will configure the bond with the specified options. The options
specified in BONDING_OPTS are identical to the bonding module parameters
except for the arp_ip_target field when using versions of initscripts older
than and 8.57 (Fedora 8) and 8.45.19 (Red Hat Enterprise Linux 5.2). When
using older versions each target should be included as a separate option and
should be preceded by a '+' to indicate it should be added to the list of
queried targets, e.g.,::
arp_ip_target=+192.168.1.1 arp_ip_target=+192.168.1.2
is the proper syntax to specify multiple targets. When specifying
options via BONDING_OPTS, it is not necessary to edit
``/etc/modprobe.d/*.conf``.
For even older versions of initscripts that do not support
BONDING_OPTS, it is necessary to edit /etc/modprobe.d/*.conf, depending upon
your distro) to load the bonding module with your desired options when the
bond0 interface is brought up. The following lines in /etc/modprobe.d/*.conf
will load the bonding module, and select its options:
alias bond0 bonding
options bond0 mode=balance-alb miimon=100
Replace the sample parameters with the appropriate set of
options for your configuration.
Finally run "/etc/rc.d/init.d/network restart" as root. This
will restart the networking subsystem and your bond link should be now
up and running.
3.2.1 Using DHCP with Initscripts
---------------------------------
Recent versions of initscripts (the versions supplied with Fedora
Core 3 and Red Hat Enterprise Linux 4, or later versions, are reported to
work) have support for assigning IP information to bonding devices via
DHCP.
To configure bonding for DHCP, configure it as described
above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp"
and add a line consisting of "TYPE=Bonding". Note that the TYPE value
is case sensitive.
3.2.2 Configuring Multiple Bonds with Initscripts
-------------------------------------------------
Initscripts packages that are included with Fedora 7 and Red Hat
Enterprise Linux 5 support multiple bonding interfaces by simply
specifying the appropriate BONDING_OPTS= in ifcfg-bondX where X is the
number of the bond. This support requires sysfs support in the kernel,
and a bonding driver of version 3.0.0 or later. Other configurations may
not support this method for specifying multiple bonding interfaces; for
those instances, see the "Configuring Multiple Bonds Manually" section,
below.
Iproute2 수동 구성과 여러 instance
1387-15033.3 Iproute2 수동 구성은 init script가 bonding을 모르는 distribution에 적용됩니다. Module parameter를 `/etc/modprobe.d/`에 두고 sysconfig의 `/etc/init.d/boot.local` 또는 initscripts의 `/etc/rc.d/rc.local`에 modprobe·`ip link` 명령을 추가합니다.
두 e100 interface를 balance-alb·miimon 100으로 영구 구성하는 예는 bonding과 e100을 load하고 bond0에 IP를 올린 뒤 `ip link set eth0 master bond0`, `ip link set eth1 master bond0`을 실행합니다. 이 방식은 bond의 ifup·ifdown을 지원하지 않아 init script나 bonding 전용 별도 script를 다시 실행해야 합니다.
종료는 bond0를 down하고 bonding과 slave driver module을 제거합니다.
3.3.1 같은 option의 여러 bond는 `max_bonds`를 쓸 수 있지만 서로 다른 option이라면 sysfs가 바람직합니다. Sysfs가 없는 옛 bonding은 driver를 서로 다른 module 이름으로 여러 번 load해야 했습니다.
Modprobe config에서 bond0 instance를 `-o bond0 mode=balance-rr miimon=100`, bond1을 `-o bond1 mode=balance-alb miimon=50`처럼 별명·고유 이름으로 load합니다. 두 번째 option을 보지 못하는 옛 환경은 `install bond1 /sbin/modprobe --ignore-install bonding -o bond1 ...` 형식을 사용합니다.
일부 Fedora·RHEL 4 kernel은 module load 시 `-o bond1` rename을 `Operation not permitted`로 거부합니다. 이런 kernel은 sysfs도 없어 서로 다른 parameter의 여러 bond를 구성할 수 없습니다.
3.3 Configuring Bonding Manually with iproute2
-----------------------------------------------
This section applies to distros whose network initialization
scripts (the sysconfig or initscripts package) do not have specific
knowledge of bonding. One such distro is SuSE Linux Enterprise Server
version 8.
The general method for these systems is to place the bonding
module parameters into a config file in /etc/modprobe.d/ (as
appropriate for the installed distro), then add modprobe and/or
`ip link` commands to the system's global init script. The name of
the global init script differs; for sysconfig, it is
/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
For example, if you wanted to make a simple bond of two e100
devices (presumed to be eth0 and eth1), and have it persist across
reboots, edit the appropriate file (/etc/init.d/boot.local or
/etc/rc.d/rc.local), and add the following::
modprobe bonding mode=balance-alb miimon=100
modprobe e100
ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
ip link set eth0 master bond0
ip link set eth1 master bond0
Replace the example bonding module parameters and bond0
network configuration (IP address, netmask, etc) with the appropriate
values for your configuration.
Unfortunately, this method will not provide support for the
ifup and ifdown scripts on the bond devices. To reload the bonding
configuration, it is necessary to run the initialization script, e.g.,::
# /etc/init.d/boot.local
or::
# /etc/rc.d/rc.local
It may be desirable in such a case to create a separate script
which only initializes the bonding configuration, then call that
separate script from within boot.local. This allows for bonding to be
enabled without re-running the entire global init script.
To shut down the bonding devices, it is necessary to first
mark the bonding device itself as being down, then remove the
appropriate device driver modules. For our example above, you can do
the following::
# ifconfig bond0 down
# rmmod bonding
# rmmod e100
Again, for convenience, it may be desirable to create a script
with these commands.
3.3.1 Configuring Multiple Bonds Manually
-----------------------------------------
This section contains information on configuring multiple
bonding devices with differing options for those systems whose network
initialization scripts lack support for configuring multiple bonds.
If you require multiple bonding devices, but all with the same
options, you may wish to use the "max_bonds" module parameter,
documented above.
To create multiple bonding devices with differing options, it is
preferable to use bonding parameters exported by sysfs, documented in the
section below.
For versions of bonding without sysfs support, the only means to
provide multiple instances of bonding with differing options is to load
the bonding driver multiple times. Note that current versions of the
sysconfig network initialization scripts handle this automatically; if
your distro uses these scripts, no special action is needed. See the
section Configuring Bonding Devices, above, if you're not sure about your
network initialization scripts.
To load multiple instances of the module, it is necessary to
specify a different name for each instance (the module loading system
requires that every loaded module, even multiple instances of the same
module, have a unique name). This is accomplished by supplying multiple
sets of bonding options in ``/etc/modprobe.d/*.conf``, for example::
alias bond0 bonding
options bond0 -o bond0 mode=balance-rr miimon=100
alias bond1 bonding
options bond1 -o bond1 mode=balance-alb miimon=50
will load the bonding module two times. The first instance is
named "bond0" and creates the bond0 device in balance-rr mode with an
miimon of 100. The second instance is named "bond1" and creates the
bond1 device in balance-alb mode with an miimon of 50.
In some circumstances (typically with older distributions),
the above does not work, and the second bonding instance never sees
its options. In that case, the second options line can be substituted
as follows::
install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \
mode=balance-alb miimon=50
This may be repeated any number of times, specifying a new and
unique name in place of bond1 for each subsequent instance.
It has been observed that some Red Hat supplied kernels are unable
to rename modules at load time (the "-o bond1" part). Attempts to pass
that option to modprobe will produce an "Operation not permitted" error.
This has been reported on some Fedora Core kernels, and has been seen on
RHEL 4 as well. On kernels exhibiting this problem, it will be impossible
to configure multiple bonds with differing parameters (as they are older
kernels, and also lack sysfs support).
Sysfs runtime 구성
1504-16573.4 Bonding 3.0.0부터 sysfs로 모든 bond를 module unload 없이 동적 구성하고 runtime에 추가·삭제할 수 있습니다. `ifenslave`는 더 이상 필요하지 않지만 계속 지원됩니다. Kernel built-in bonding에서도 서로 다른 여러 bond를 구성할 수 있으며 sysfs가 `/sys`에 mount되어 있어야 합니다.
Bond 생성·삭제: `echo +foo > /sys/class/net/bonding_masters`, `echo -bar > ...`, 목록은 `cat`으로 봅니다. Sysfs file 4KiB 제한 때문에 수백 개 이상이면 목록이 잘릴 수 있습니다.
Slave 추가·삭제: `/sys/class/net/<bond>/bonding/slaves`에 `+eth0`, `-eth0`를 씁니다. Enslave하면 bond의 `slave_eth0`와 slave의 `master` symlink가 생깁니다. `/sys/class/net/eth0/master/bonding/slaves`에 `-eth0`를 쓰면 bond 이름을 몰라도 분리할 수 있습니다.
각 `/sys/class/net/<bond>/bonding` file은 command-line parameter와 같은 이름·값을 받으며 `arp_ip_target`만 추가·삭제 syntax가 다릅니다. Mode 변경 전 bond를 down해야 합니다. `mode`에 6 또는 `balance-alb`, `miimon`에 1000을 쓸 수 있습니다. MII를 켜면 ARP monitoring이 꺼지고 그 반대도 같습니다.
ARP target은 `+192.168.0.100`으로 최대 16개 추가하고 `-192.168.0.100`으로 제거합니다. `lp_interval`은 learning packet 사이 초이며 기본 1입니다.
영구 예시는 module·driver를 load하고 mode, IP, miimon을 정한 뒤 `+eth0`, `+eth1`을 slaves에 씁니다. 두 번째 bond는 bonding_masters에 `+bond1`, active-backup mode, IP, ARP target·interval, `+eth2`, `+eth3`을 설정합니다.
3.4 Configuring Bonding Manually via Sysfs
------------------------------------------
Starting with version 3.0.0, Channel Bonding may be configured
via the sysfs interface. This interface allows dynamic configuration
of all bonds in the system without unloading the module. It also
allows for adding and removing bonds at runtime. Ifenslave is no
longer required, though it is still supported.
Use of the sysfs interface allows you to use multiple bonds
with different configurations without having to reload the module.
It also allows you to use multiple, differently configured bonds when
bonding is compiled into the kernel.
You must have the sysfs filesystem mounted to configure
bonding this way. The examples in this document assume that you
are using the standard mount point for sysfs, e.g. /sys. If your
sysfs filesystem is mounted elsewhere, you will need to adjust the
example paths accordingly.
Creating and Destroying Bonds
-----------------------------
To add a new bond foo::
# echo +foo > /sys/class/net/bonding_masters
To remove an existing bond bar::
# echo -bar > /sys/class/net/bonding_masters
To show all existing bonds::
# cat /sys/class/net/bonding_masters
.. note::
due to 4K size limitation of sysfs files, this list may be
truncated if you have more than a few hundred bonds. This is unlikely
to occur under normal operating conditions.
Adding and Removing Slaves
--------------------------
Interfaces may be enslaved to a bond using the file
/sys/class/net/<bond>/bonding/slaves. The semantics for this file
are the same as for the bonding_masters file.
To enslave interface eth0 to bond bond0::
# ifconfig bond0 up
# echo +eth0 > /sys/class/net/bond0/bonding/slaves
To free slave eth0 from bond bond0::
# echo -eth0 > /sys/class/net/bond0/bonding/slaves
When an interface is enslaved to a bond, symlinks between the
two are created in the sysfs filesystem. In this case, you would get
/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and
/sys/class/net/eth0/master pointing to /sys/class/net/bond0.
This means that you can tell quickly whether or not an
interface is enslaved by looking for the master symlink. Thus:
# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves
will free eth0 from whatever bond it is enslaved to, regardless of
the name of the bond interface.
Changing a Bond's Configuration
-------------------------------
Each bond may be configured individually by manipulating the
files located in /sys/class/net/<bond name>/bonding
The names of these files correspond directly with the command-
line parameters described elsewhere in this file, and, with the
exception of arp_ip_target, they accept the same values. To see the
current setting, simply cat the appropriate file.
A few examples will be given here; for specific usage
guidelines for each parameter, see the appropriate section in this
document.
To configure bond0 for balance-alb mode::
# ifconfig bond0 down
# echo 6 > /sys/class/net/bond0/bonding/mode
- or -
# echo balance-alb > /sys/class/net/bond0/bonding/mode
.. note::
The bond interface must be down before the mode can be changed.
To enable MII monitoring on bond0 with a 1 second interval::
# echo 1000 > /sys/class/net/bond0/bonding/miimon
.. note::
If ARP monitoring is enabled, it will disabled when MII
monitoring is enabled, and vice-versa.
To add ARP targets::
# echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
# echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target
.. note::
up to 16 target addresses may be specified.
To remove an ARP target::
# echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
To configure the interval between learning packet transmits::
# echo 12 > /sys/class/net/bond0/bonding/lp_interval
.. note::
the lp_interval is the number of seconds between instances where
the bonding driver sends learning packets to each slaves peer switch. The
default interval is 1 second.
Example Configuration
---------------------
We begin with the same example that is shown in section 3.3,
executed with sysfs, and without using ifenslave.
To make a simple bond of two e100 devices (presumed to be eth0
and eth1), and have it persist across reboots, edit the appropriate
file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the
following::
modprobe bonding
modprobe e100
echo balance-alb > /sys/class/net/bond0/bonding/mode
ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
echo 100 > /sys/class/net/bond0/bonding/miimon
echo +eth0 > /sys/class/net/bond0/bonding/slaves
echo +eth1 > /sys/class/net/bond0/bonding/slaves
To add a second bond, with two e1000 interfaces in
active-backup mode, using ARP monitoring, add the following lines to
your init script::
modprobe e1000
echo +bond1 > /sys/class/net/bonding_masters
echo active-backup > /sys/class/net/bond1/bonding/mode
ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up
echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target
echo 2000 > /sys/class/net/bond1/bonding/arp_interval
echo +eth2 > /sys/class/net/bond1/bonding/slaves
echo +eth3 > /sys/class/net/bond1/bonding/slaves
Debian interfaces와 queue override
1658-17913.5 `/etc/network/interfaces`를 쓰는 Debian 계열은 `ifenslave-2.6` package로 `bond-*` option을 제공합니다. Package가 bonding module을 load하고 필요할 때 ifenslave를 실행합니다.
일반 stanza는 bond0 DHCP, `bond-slaves eth0 eth1`, active-backup, miimon 100, primary 순서를 지정합니다. Upstart system에서는 bond의 `bond-slaves none`과 각 eth interface의 manual stanza·`bond-master bond0`를 따로 둡니다. 전체 option과 advanced 예시는 `/usr/share/doc/ifenslave-2.6`을 봅니다.
3.6 특수 policy override: 보통 physical 송신 port는 mode policy가 고르지만 특정 traffic class를 특정 interface에 우선 보내고 fallback을 둘 수 있습니다. Linux traffic control로 이를 구현합니다.
Bonding은 기본적으로 multiqueue-aware이며 초기화 때 16 queue를 만듭니다. Module parameter `tx_queues`로만 바꾸며 allocation이 module init 시라 sysfs parameter는 없습니다.
`/proc/net/bonding/bondX`는 slave마다 queue ID를 출력합니다. `echo "eth1:2" > /sys/class/net/bond0/bonding/queue_id`처럼 qid를 정하고 필요한 모든 slave에 반복합니다. Initscripts는 여러 `queue_id`를 BONDING_OPTS에 넣을 수 있습니다.
TC `multiq` qdisc와 u32 filter의 `skbedit queue_mapping 2`를 사용하면 destination `192.168.1.100` traffic을 qid 2에 대응하는 eth1로 강제합니다. Qid는 1부터 시작하고 0은 정상 mode policy 선택을 뜻합니다. Slave qid 0은 slave 자체 TC queue selection을 pass-through할 수 있습니다. 이 기능은 bonding 3.7.0에 처음 등장했고 초기에는 round-robin·active-backup만 output slave override를 지원했습니다.
3.5 Configuration with Interfaces Support
-----------------------------------------
This section applies to distros which use /etc/network/interfaces file
to describe network interface configuration, most notably Debian and its
derivatives.
The ifup and ifdown commands on Debian don't support bonding out of
the box. The ifenslave-2.6 package should be installed to provide bonding
support. Once installed, this package will provide ``bond-*`` options
to be used into /etc/network/interfaces.
Note that ifenslave-2.6 package will load the bonding module and use
the ifenslave command when appropriate.
Example Configurations
----------------------
In /etc/network/interfaces, the following stanza will configure bond0, in
active-backup mode, with eth0 and eth1 as slaves::
auto bond0
iface bond0 inet dhcp
bond-slaves eth0 eth1
bond-mode active-backup
bond-miimon 100
bond-primary eth0 eth1
If the above configuration doesn't work, you might have a system using
upstart for system startup. This is most notably true for recent
Ubuntu versions. The following stanza in /etc/network/interfaces will
produce the same result on those systems::
auto bond0
iface bond0 inet dhcp
bond-slaves none
bond-mode active-backup
bond-miimon 100
auto eth0
iface eth0 inet manual
bond-master bond0
bond-primary eth0 eth1
auto eth1
iface eth1 inet manual
bond-master bond0
bond-primary eth0 eth1
For a full list of ``bond-*`` supported options in /etc/network/interfaces and
some more advanced examples tailored to you particular distros, see the files in
/usr/share/doc/ifenslave-2.6.
3.6 Overriding Configuration for Special Cases
----------------------------------------------
When using the bonding driver, the physical port which transmits a frame is
typically selected by the bonding driver, and is not relevant to the user or
system administrator. The output port is simply selected using the policies of
the selected bonding mode. On occasion however, it is helpful to direct certain
classes of traffic to certain physical interfaces on output to implement
slightly more complex policies. For example, to reach a web server over a
bonded interface in which eth0 connects to a private network, while eth1
connects via a public network, it may be desirous to bias the bond to send said
traffic over eth0 first, using eth1 only as a fall back, while all other traffic
can safely be sent over either interface. Such configurations may be achieved
using the traffic control utilities inherent in linux.
By default the bonding driver is multiqueue aware and 16 queues are created
when the driver initializes (see Documentation/networking/multiqueue.rst
for details). If more or less queues are desired the module parameter
tx_queues can be used to change this value. There is no sysfs parameter
available as the allocation is done at module init time.
The output of the file /proc/net/bonding/bondX has changed so the output Queue
ID is now printed for each slave::
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1a:a0:12:8f:cb
Slave queue ID: 0
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1a:a0:12:8f:cc
Slave queue ID: 2
The queue_id for a slave can be set using the command::
# echo "eth1:2" > /sys/class/net/bond0/bonding/queue_id
Any interface that needs a queue_id set should set it with multiple calls
like the one above until proper priorities are set for all interfaces. On
distributions that allow configuration via initscripts, multiple 'queue_id'
arguments can be added to BONDING_OPTS to set all needed slave queues.
These queue id's can be used in conjunction with the tc utility to configure
a multiqueue qdisc and filters to bias certain traffic to transmit on certain
slave devices. For instance, say we wanted, in the above configuration to
force all traffic bound to 192.168.1.100 to use eth1 in the bond as its output
device. The following commands would accomplish this::
# tc qdisc add dev bond0 handle 1 root multiq
# tc filter add dev bond0 protocol ip parent 1: prio 1 u32 match ip \
dst 192.168.1.100 action skbedit queue_mapping 2
These commands tell the kernel to attach a multiqueue queue discipline to the
bond0 interface and filter traffic enqueued to it, such that packets with a dst
ip of 192.168.1.100 have their output queue mapping value overwritten to 2.
This value is then passed into the driver, causing the normal output path
selection policy to be overridden, selecting instead qid 2, which maps to eth1.
Note that qid values begin at 1. Qid 0 is reserved to initiate to the driver
that normal output policy selection should take place. One benefit to simply
leaving the qid for a slave to 0 is the multiqueue awareness in the bonding
driver that is now present. This awareness allows tc filters to be placed on
slave devices as well as bond devices and the bonding driver will simply act as
a pass-through for selecting output queues on the slave device rather than
output port selection.
This feature first appeared in bonding driver version 3.7.0 and support for
output slave selection was limited to round-robin and active-backup modes.
더 안전한 LACP actor 식별자
1792-18383.7 802.3ad mode에서 host actor와 switch partner는 link-local MAC 대상 LACPDU를 교환하므로 일반 bridge가 전달하지 않아 sniff하기 어렵습니다. 하지만 값 대부분이 예측 가능하거나 host MAC 그대로여서 같은 L2의 다른 system이 LACPDU를 spoof해 다른 host aggregate에 합류하고 수신 traffic 일부를 받거나 그 host를 사칭할 가능성이 있습니다.
가능성은 낮지만 세 parameter를 randomize해 완화할 수 있습니다. `ad_actor_system`에는 NULL·multicast가 아니고 local-admin bit를 켠 random MAC을 shell의 `$RANDOM`으로 만들어 sysfs에 씁니다.
`ad_actor_sys_prio`는 기본 65535 대신 1~65535 random system priority를 설정합니다. `ad_user_port_key`는 비어 있는 기본 상위 10 bit 대신 `RANDOM & 0x3FF`로 0~1023 값을 정합니다. 원문의 정확한 shell command를 보존합니다.
3.7 Configuring LACP for 802.3ad mode in a more secure way
----------------------------------------------------------
When using 802.3ad bonding mode, the Actor (host) and Partner (switch)
exchange LACPDUs. These LACPDUs cannot be sniffed, because they are
destined to link local mac addresses (which switches/bridges are not
supposed to forward). However, most of the values are easily predictable
or are simply the machine's MAC address (which is trivially known to all
other hosts in the same L2). This implies that other machines in the L2
domain can spoof LACPDU packets from other hosts to the switch and potentially
cause mayhem by joining (from the point of view of the switch) another
machine's aggregate, thus receiving a portion of that hosts incoming
traffic and / or spoofing traffic from that machine themselves (potentially
even successfully terminating some portion of flows). Though this is not
a likely scenario, one could avoid this possibility by simply configuring
few bonding parameters:
(a) ad_actor_system : You can set a random mac-address that can be used for
these LACPDU exchanges. The value can not be either NULL or Multicast.
Also it's preferable to set the local-admin bit. Following shell code
generates a random mac-address as described above::
# sys_mac_addr=$(printf '%02x:%02x:%02x:%02x:%02x:%02x' \
$(( (RANDOM & 0xFE) | 0x02 )) \
$(( RANDOM & 0xFF )) \
$(( RANDOM & 0xFF )) \
$(( RANDOM & 0xFF )) \
$(( RANDOM & 0xFF )) \
$(( RANDOM & 0xFF )))
# echo $sys_mac_addr > /sys/class/net/bond0/bonding/ad_actor_system
(b) ad_actor_sys_prio : Randomize the system priority. The default value
is 65535, but system can take the value from 1 - 65535. Following shell
code generates random priority and sets it::
# sys_prio=$(( 1 + RANDOM + RANDOM ))
# echo $sys_prio > /sys/class/net/bond0/bonding/ad_actor_sys_prio
(c) ad_user_port_key : Use the user portion of the port-key. The default
keeps this empty. These are the upper 10 bits of the port-key and value
ranges from 0 - 1023. Following shell code generates these 10 bits and
sets it::
# usr_port_key=$(( RANDOM & 0x3FF ))
# echo $usr_port_key > /sys/class/net/bond0/bonding/ad_user_port_key
Bonding 상태 조회
1839-19064. Bonding 구성 조회
각 bond에는 `/proc/net/bonding` 아래 read-only file이 있고 mode·option과 각 slave 상태를 담습니다. Mode 0, miimon 1000 예시는 driver version, round-robin mode, current active, MII 상태·주기, up/down delay, slave별 상태와 link failure count를 표시합니다. 정확한 형식은 driver version과 구성·상태에 따라 바뀝니다.
Network 설정은 `ifconfig`로 확인합니다. Bond device는 MASTER, slave는 SLAVE flag가 있지만 어느 slave가 어느 master에 속하는지는 ifconfig만으로 알 수 없습니다.
TLB·ALB처럼 slave별 고유 MAC이 필요한 mode를 제외하면 bond0와 모든 slave가 같은 HWaddr을 가집니다. 원문은 bond0, eth0, eth1의 packet counter, MTU, interrupt와 base address가 포함된 전체 출력 예를 보존합니다.
4 Querying Bonding Configuration
=================================
4.1 Bonding Configuration
-------------------------
Each bonding device has a read-only file residing in the
/proc/net/bonding directory. The file contents include information
about the bonding configuration, options and state of each slave.
For example, the contents of /proc/net/bonding/bond0 after the
driver is loaded with parameters of mode=0 and miimon=1000 is
generally as follows::
Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004)
Bonding Mode: load balancing (round-robin)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 1000
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Link Failure Count: 1
Slave Interface: eth0
MII Status: up
Link Failure Count: 1
The precise format and contents will change depending upon the
bonding configuration, state, and version of the bonding driver.
4.2 Network configuration
-------------------------
The network configuration can be inspected using the ifconfig
command. Bonding devices will have the MASTER flag set; Bonding slave
devices will have the SLAVE flag set. The ifconfig output does not
contain information on which slaves are associated with which masters.
In the example below, the bond0 interface is the master
(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of
bond0 have the same MAC address (HWaddr) as bond0 for all modes except
TLB and ALB that require a unique MAC address for each slave::
# /sbin/ifconfig
bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
collisions:0 txqueuelen:0
eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
collisions:0 txqueuelen:100
Interrupt:10 Base address:0x1080
eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
Interrupt:9 Base address:0x1400
Switch 구성과 802.1Q VLAN
1907-19905. 여기서 switch는 bonded device cable 반대편의 직접 연결 system이며 전용 switch 또는 다른 Linux host일 수 있습니다. Active-backup, balance-tlb, balance-alb는 특별한 switch 구성이 필요 없습니다.
802.3ad는 switch port를 802.3ad aggregation으로 묶어야 합니다. 예를 들어 Cisco 3550은 port를 한 EtherChannel instance로 묶고 mode를 `lacp`로 설정합니다. Balance-rr, balance-xor, broadcast도 보통 port를 EtherChannel·trunk group 등으로 묶어야 하며 switch의 송신 hash는 MAC 또는 IP XOR 등을 고릅니다. 양 peer의 transmit policy가 같을 필요는 없습니다.
6. 802.1Q VLAN
8021q driver로 bond 위에 VLAN device를 만들 수 있습니다. 8021q를 거친 packet은 자동 tag되지만 bonding learning packet, ALB ARP, ARP monitor probe 같은 self-generated packet은 bonding이 직접 tag합니다. 따라서 bonding은 위에 구성된 VLAN ID를 학습해야 합니다.
Bond는 VLAN hardware offload가 완전히 가능한 것으로 선언해 `add_vid`·`kill_vid` notification을 받고 slave에 전파합니다. Offload 가능·불가능 adapter가 섞이면 offload 불가 slave로 나가는 accelerated packet을 bonding이 un-accelerate해 tag를 일반 위치에 놓습니다.
VLAN interface는 최소 한 slave를 bond에 추가한 뒤 만들어야 합니다. 첫 slave 전 bond MAC은 all-zero이므로 먼저 VLAN을 만들면 잘못된 MAC을 복사합니다. 모든 slave를 제거한 채 VLAN이 남아 있다가 새 slave를 넣으면 VLAN의 옛 MAC과 bond의 새 MAC이 다를 수 있습니다.
해결책은 모든 VLAN interface를 제거·재생성하거나 bond MAC을 VLAN MAC과 맞추는 것입니다. VLAN interface MAC을 바꾸면 underlying bond가 promiscuous mode가 될 수 있습니다.
5. Switch Configuration
=======================
For this section, "switch" refers to whatever system the
bonded devices are directly connected to (i.e., where the other end of
the cable plugs into). This may be an actual dedicated switch device,
or it may be another regular system (e.g., another computer running
Linux),
The active-backup, balance-tlb and balance-alb modes do not
require any specific configuration of the switch.
The 802.3ad mode requires that the switch have the appropriate
ports configured as an 802.3ad aggregation. The precise method used
to configure this varies from switch to switch, but, for example, a
Cisco 3550 series switch requires that the appropriate ports first be
grouped together in a single etherchannel instance, then that
etherchannel is set to mode "lacp" to enable 802.3ad (instead of
standard EtherChannel).
The balance-rr, balance-xor and broadcast modes generally
require that the switch have the appropriate ports grouped together.
The nomenclature for such a group differs between switches, it may be
called an "etherchannel" (as in the Cisco example, above), a "trunk
group" or some other similar variation. For these modes, each switch
will also have its own configuration options for the switch's transmit
policy to the bond. Typical choices include XOR of either the MAC or
IP addresses. The transmit policy of the two peers does not need to
match. For these three modes, the bonding mode really selects a
transmit policy for an EtherChannel group; all three will interoperate
with another EtherChannel group.
6. 802.1q VLAN Support
======================
It is possible to configure VLAN devices over a bond interface
using the 8021q driver. However, only packets coming from the 8021q
driver and passing through bonding will be tagged by default. Self
generated packets, for example, bonding's learning packets or ARP
packets generated by either ALB mode or the ARP monitor mechanism, are
tagged internally by bonding itself. As a result, bonding must
"learn" the VLAN IDs configured above it, and use those IDs to tag
self generated packets.
For reasons of simplicity, and to support the use of adapters
that can do VLAN hardware acceleration offloading, the bonding
interface declares itself as fully hardware offloading capable, it gets
the add_vid/kill_vid notifications to gather the necessary
information, and it propagates those actions to the slaves. In case
of mixed adapter types, hardware accelerated tagged packets that
should go through an adapter that is not offloading capable are
"un-accelerated" by the bonding driver so the VLAN tag sits in the
regular location.
VLAN interfaces *must* be added on top of a bonding interface
only after enslaving at least one slave. The bonding interface has a
hardware address of 00:00:00:00:00:00 until the first slave is added.
If the VLAN interface is created prior to the first enslavement, it
would pick up the all-zeroes hardware address. Once the first slave
is attached to the bond, the bond device itself will pick up the
slave's hardware address, which is then available for the VLAN device.
Also, be aware that a similar problem can occur if all slaves
are released from a bond that still has one or more VLAN interfaces on
top of it. When a new slave is added, the bonding interface will
obtain its hardware address from the first slave, which might not
match the hardware address of the VLAN interfaces (which was
ultimately copied from an earlier slave).
There are two methods to ensure that the VLAN device operates
with the correct hardware address if all slaves are removed from a
bond interface:
1. Remove all VLAN interfaces then recreate them
2. Set the bonding interface's hardware address so that it
matches the hardware address of the VLAN interfaces.
Note that changing a VLAN interface's HW address would set the
underlying device -- i.e. the bonding interface -- to promiscuous
mode, which might not be what you want.
ARP·MII link monitoring
1991-20457. Bonding driver는 slave link 상태를 ARP monitor와 MII monitor 두 방식으로 감시합니다. 구현 제약으로 둘을 동시에 활성화할 수 없습니다.
7.1 ARP monitor는 지정한 local-network peer 하나 이상에 ARP query를 보내 reply로 실제 왕복 traffic이 흐르는지 확인합니다.
7.2 HA에서는 한 target 자체의 failure를 link failure로 오인하지 않도록 여러 target을 쓰는 편이 안정적입니다. Module option에서는 comma로 `arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9`처럼 나열합니다.
7.3 MII monitor는 local interface carrier state만 봅니다. Device driver의 carrier 유지, MII register query, ethtool query 중 하나로 확인하며 현재는 `netif_carrier` subsystem이 제공하는 driver carrier 정보를 사용합니다.
7. Link Monitoring
==================
The bonding driver at present supports two schemes for
monitoring a slave device's link state: the ARP monitor and the MII
monitor.
At the present time, due to implementation restrictions in the
bonding driver itself, it is not possible to enable both ARP and MII
monitoring simultaneously.
7.1 ARP Monitor Operation
-------------------------
The ARP monitor operates as its name suggests: it sends ARP
queries to one or more designated peer systems on the network, and
uses the response as an indication that the link is operating. This
gives some assurance that traffic is actually flowing to and from one
or more peers on the local network.
7.2 Configuring Multiple ARP Targets
------------------------------------
While ARP monitoring can be done with just one target, it can
be useful in a High Availability setup to have several targets to
monitor. In the case of just one target, the target itself may go
down or have a problem making it unresponsive to ARP requests. Having
an additional target (or several) increases the reliability of the ARP
monitoring.
Multiple ARP targets must be separated by commas as follows::
# example options for ARP monitoring with three targets
alias bond0 bonding
options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
For just a single target the options would resemble::
# example options for ARP monitoring with one target
alias bond0 bonding
options bond0 arp_interval=60 arp_ip_target=192.168.0.100
7.3 MII Monitor Operation
-------------------------
The MII monitor monitors only the carrier state of the local
network interface. It accomplishes this in one of three ways: by
depending upon the device driver to maintain its carrier state, by
querying the device's MII registers, or by making an ethtool query to
the device.
The MII monitor relies on the driver for carrier state information (via
the netif_carrier subsystem).
Routing과 interface 이름 문제
2046-21278.1 Bonding을 구성할 때 slave는 master보다 우선하는 route를 가져서는 안 되며 일반적으로 route가 없어야 합니다. Eth0·eth1·bond0 모두 같은 10.0.0.0/16 route를 가지면 outgoing traffic이 bond0 대신 slave로 나가 driver를 우회할 수 있습니다.
ARP monitor request는 bond0로 나갔지만 reply가 eth0로 들어오면 interface 단위로 matching하는 ARP layer는 unsolicited reply로 보고 버립니다. Slave route를 제거하거나 master route보다 우선하지 않게 해야 합니다. MII monitor는 routing table 영향을 받지 않습니다.
8.2 Physical device와 ethX 이름을 고정하지 않는 system에서는 driver load 순서 때문에 interface 이름이 바뀔 수 있습니다. Bonding이 먼저 load되고 e1000이 다음이면 예상 eth2·eth3 대신 eth0·eth1을 가져 slave 구성이 어긋납니다.
옛 modules.conf는 `add above bonding e1000 tg3`로 순서를 고정합니다. Modprobe system은 `/etc/modprobe.d/`의 `softdep bonding pre: tg3 e1000`으로 network driver를 bonding보다 먼저 load합니다.
8. Potential Sources of Trouble
===============================
8.1 Adventures in Routing
-------------------------
When bonding is configured, it is important that the slave
devices not have routes that supersede routes of the master (or,
generally, not have routes at all). For example, suppose the bonding
device bond0 has two slaves, eth0 and eth1, and the routing table is
as follows::
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1
10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0
127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo
This routing configuration will likely still update the
receive/transmit times in the driver (needed by the ARP monitor), but
may bypass the bonding driver (because outgoing traffic to, in this
case, another host on network 10 would use eth0 or eth1 before bond0).
The ARP monitor (and ARP itself) may become confused by this
configuration, because ARP requests (generated by the ARP monitor)
will be sent on one interface (bond0), but the corresponding reply
will arrive on a different interface (eth0). This reply looks to ARP
as an unsolicited ARP reply (because ARP matches replies on an
interface basis), and is discarded. The MII monitor is not affected
by the state of the routing table.
The solution here is simply to ensure that slaves do not have
routes of their own, and if for some reason they must, those routes do
not supersede routes of their master. This should generally be the
case, but unusual configurations or errant manual or automatic static
route additions may cause trouble.
8.2 Ethernet Device Renaming
----------------------------
On systems with network configuration scripts that do not
associate physical devices directly with network interface names (so
that the same physical device always has the same "ethX" name), it may
be necessary to add some special logic to config files in
/etc/modprobe.d/.
For example, given a modules.conf containing the following::
alias bond0 bonding
options bond0 mode=some-mode miimon=50
alias eth0 tg3
alias eth1 tg3
alias eth2 e1000
alias eth3 e1000
If neither eth0 and eth1 are slaves to bond0, then when the
bond0 interface comes up, the devices may end up reordered. This
happens because bonding is loaded first, then its slave device's
drivers are loaded next. Since no other drivers have been loaded,
when the e1000 driver loads, it will receive eth0 and eth1 for its
devices, but the bonding configuration tries to enslave eth2 and eth3
(which may later be assigned to the tg3 devices).
Adding the following::
add above bonding e1000 tg3
causes modprobe to load e1000 then tg3, in that order, when
bonding is loaded. This command is fully documented in the
modules.conf manual page.
On systems utilizing modprobe an equivalent problem can occur.
In this case, the following can be added to config files in
/etc/modprobe.d/ as::
softdep bonding pre: tg3 e1000
This will load tg3 and e1000 modules before loading the bonding one.
Full documentation on this can be found in the modprobe.d and modprobe
manual pages.
SNMP interface index와 promiscuous mode
2128-22029. SNMP agent를 쓰면 bond member network driver보다 bonding driver를 먼저 load해야 합니다. IP별 `ipAdEntIfIndex`는 해당 IP를 가진 첫 interface에 연결되므로 eth0 driver가 먼저면 bond IP가 eth0 index에 잘못 연결됩니다. Bonding을 먼저 load하면 bond0가 앞 index를 얻고 IP가 올바르게 매핑됩니다. `ifDescr`에 이름을 표시하지 않는 distribution에서도 IP와 IfIndex 연관은 남습니다.
10. Tcpdump 같은 monitor는 모든 traffic을 보려고 promiscuous mode를 켭니다. Bonding은 master의 변경을 slave로 전파합니다.
Balance-rr, balance-xor, broadcast, 802.3ad는 모든 slave에 전파합니다. Active-backup, balance-tlb, balance-alb는 active slave에만 전파하고 failover 시 새 active에 적용합니다. TLB의 active는 inbound를 받는 slave, ALB의 active는 mode control traffic과 미할당 peer에 쓰는 primary입니다.
9. SNMP agents
===============
If running SNMP agents, the bonding driver should be loaded
before any network drivers participating in a bond. This requirement
is due to the interface index (ipAdEntIfIndex) being associated to
the first interface found with a given IP address. That is, there is
only one ipAdEntIfIndex for each IP address. For example, if eth0 and
eth1 are slaves of bond0 and the driver for eth0 is loaded before the
bonding driver, the interface for the IP address will be associated
with the eth0 interface. This configuration is shown below, the IP
address 192.168.1.1 has an interface index of 2 which indexes to eth0
in the ifDescr table (ifDescr.2).
::
interfaces.ifTable.ifEntry.ifDescr.1 = lo
interfaces.ifTable.ifEntry.ifDescr.2 = eth0
interfaces.ifTable.ifEntry.ifDescr.3 = eth1
interfaces.ifTable.ifEntry.ifDescr.4 = eth2
interfaces.ifTable.ifEntry.ifDescr.5 = eth3
interfaces.ifTable.ifEntry.ifDescr.6 = bond0
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
This problem is avoided by loading the bonding driver before
any network drivers participating in a bond. Below is an example of
loading the bonding driver first, the IP address 192.168.1.1 is
correctly associated with ifDescr.2.
interfaces.ifTable.ifEntry.ifDescr.1 = lo
interfaces.ifTable.ifEntry.ifDescr.2 = bond0
interfaces.ifTable.ifEntry.ifDescr.3 = eth0
interfaces.ifTable.ifEntry.ifDescr.4 = eth1
interfaces.ifTable.ifEntry.ifDescr.5 = eth2
interfaces.ifTable.ifEntry.ifDescr.6 = eth3
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
While some distributions may not report the interface name in
ifDescr, the association between the IP address and IfIndex remains
and SNMP functions such as Interface_Scan_Next will report that
association.
10. Promiscuous mode
====================
When running network monitoring tools, e.g., tcpdump, it is
common to enable promiscuous mode on the device, so that all traffic
is seen (instead of seeing only traffic destined for the local host).
The bonding driver handles promiscuous mode changes to the bonding
master device (e.g., bond0), and propagates the setting to the slave
devices.
For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
the promiscuous mode setting is propagated to all slaves.
For the active-backup, balance-tlb and balance-alb modes, the
promiscuous mode setting is propagated only to the active slave.
For balance-tlb mode, the active slave is the slave currently
receiving inbound traffic.
For balance-alb mode, the active slave is the slave used as a
"primary." This slave is used for mode-specific control traffic, for
sending to peers that are unassigned or if the load is unbalanced.
For the active-backup, balance-tlb and balance-alb modes, when
the active slave changes (e.g., due to a link failure), the
promiscuous setting will be propagated to the new active slave.
High Availability topology
2203-230711. HA는 host와 외부 사이에 redundant device·link·switch를 두어 다른 구성의 처리량이 더 높더라도 network가 계속 작동하도록 availability를 극대화합니다.
11.1 한 switch 또는 두 host 직접 연결 topology는 peer가 하나라 switch failure 대안이 없으므로 최대 bandwidth 최적화가 availability를 해치지 않습니다. Load-balance mode도 member link failure를 감시해 남은 link로 재분배합니다.
11.2 여러 switch에서는 usable bandwidth와 availability가 trade-off입니다. 원문 topology는 host1의 eth0·eth1이 각각 switch A·B port1에 연결되고 두 switch는 port2 ISL로 연결되며 각 port3이 외부로 나갑니다.
11.2.1 Availability 최적화에는 active-backup과 broadcast만 합리적입니다. 다른 mode는 모든 link가 같은 peer에서 끝나야 합니다. Active-backup이 일반적 권장이고 backup switch 성능·비용이 낮다면 `primary`로 선호 link를 고정합니다. Broadcast는 ISL 없이 뒤 network가 완전히 독립이고 특정 one-way traffic을 양쪽에 모두 보내야 하는 특수 경우에만 적합합니다.
11.2.2 Link monitor는 switch 능력에 좌우됩니다. Remote port3 끝 failure는 local MII로 직접 보지 못하지만 remote target ARP는 감지합니다. 여러 switch topology에서는 end-to-end failure를 잡는 ARP가 더 신뢰성 높고 switch마다 target을 하나 이상 두어야 합니다.
많은 switch의 trunk failover는 외부 port state 변화를 내부 port link state로 전파해 miimon이 감지하게 합니다. 지원·구성 방식은 switch마다 다르지만 적합한 hardware에서는 ARP monitor 대안입니다.
11. Configuring Bonding for High Availability
=============================================
High Availability refers to configurations that provide
maximum network availability by having redundant or backup devices,
links or switches between the host and the rest of the world. The
goal is to provide the maximum availability of network connectivity
(i.e., the network always works), even though other configurations
could provide higher throughput.
11.1 High Availability in a Single Switch Topology
--------------------------------------------------
If two hosts (or a host and a single switch) are directly
connected via multiple physical links, then there is no availability
penalty to optimizing for maximum bandwidth. In this case, there is
only one switch (or peer), so if it fails, there is no alternative
access to fail over to. Additionally, the bonding load balance modes
support link monitoring of their members, so if individual links fail,
the load will be rebalanced across the remaining devices.
See Section 12, "Configuring Bonding for Maximum Throughput"
for information on configuring bonding with one peer device.
11.2 High Availability in a Multiple Switch Topology
----------------------------------------------------
With multiple switches, the configuration of bonding and the
network changes dramatically. In multiple switch topologies, there is
a trade off between network availability and usable bandwidth.
Below is a sample network, configured to maximize the
availability of the network::
| |
|port3 port3|
+-----+----+ +-----+----+
| |port2 ISL port2| |
| switch A +--------------------------+ switch B |
| | | |
+-----+----+ +-----++---+
|port1 port1|
| +-------+ |
+-------------+ host1 +---------------+
eth0 +-------+ eth1
In this configuration, there is a link between the two
switches (ISL, or inter switch link), and multiple ports connecting to
the outside world ("port3" on each switch). There is no technical
reason that this could not be extended to a third switch.
11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
-------------------------------------------------------------
In a topology such as the example above, the active-backup and
broadcast modes are the only useful bonding modes when optimizing for
availability; the other modes require all links to terminate on the
same peer for them to behave rationally.
active-backup:
This is generally the preferred mode, particularly if
the switches have an ISL and play together well. If the
network configuration is such that one switch is specifically
a backup switch (e.g., has lower capacity, higher cost, etc),
then the primary option can be used to ensure that the
preferred link is always used when it is available.
broadcast:
This mode is really a special purpose mode, and is suitable
only for very specific needs. For example, if the two
switches are not connected (no ISL), and the networks beyond
them are totally independent. In this case, if it is
necessary for some specific one-way traffic to reach both
independent networks, then the broadcast mode may be suitable.
11.2.2 HA Link Monitoring Selection for Multiple Switch Topology
----------------------------------------------------------------
The choice of link monitoring ultimately depends upon your
switch. If the switch can reliably fail ports in response to other
failures, then either the MII or ARP monitors should work. For
example, in the above example, if the "port3" link fails at the remote
end, the MII monitor has no direct means to detect this. The ARP
monitor could be configured with a target at the remote end of port3,
thus detecting that failure without switch support.
In general, however, in a multiple switch topology, the ARP
monitor can provide a higher level of reliability in detecting end to
end connectivity failures (which may be caused by the failure of any
individual component to pass traffic for any reason). Additionally,
the ARP monitor should be configured with multiple targets (at least
one for each switch in the network). This will ensure that,
regardless of which switch is active, the ARP monitor has a suitable
target to query.
Note, also, that of late many switches now support a functionality
generally referred to as "trunk failover." This is a feature of the
switch that causes the link state of a particular switch port to be set
down (or up) when the state of another switch port goes down (or up).
Its purpose is to propagate link failures from logically "exterior" ports
to the logically "interior" ports that bonding is able to monitor via
miimon. Availability and configuration for trunk failover varies by
switch, but this can be a viable alternative to the ARP monitor when using
suitable switches.
Gatewayed·local throughput topology
2308-238112. 한 switch에서 최대 처리량 mode는 application과 network 환경에 따라 다릅니다. Traffic destination을 기준으로 gatewayed와 local 두 topology로 나눕니다.
Gatewayed에서는 Host A의 eth0·eth1이 router의 port1·port2에 연결되고 대부분 traffic이 router를 거쳐 다른 network의 Host B·C로 갑니다. 직접 여러 link로 연결된 두 system도 local peer가 하나라는 점에서 같습니다. 최종 destination과 관계없이 bond가 보는 MAC peer는 router 하나입니다.
Local에서는 Host A의 두 link가 switch에 붙고 Host B·C가 같은 switch의 다른 port에 있습니다. Traffic은 같은 local network의 최종 host로 직접 가므로 destination마다 고유 MAC을 봅니다.
많은 load-balancing mode가 local source·destination MAC으로 결정을 내리므로 이 구분이 중요합니다.
12. Configuring Bonding for Maximum Throughput
==============================================
12.1 Maximizing Throughput in a Single Switch Topology
------------------------------------------------------
In a single switch configuration, the best method to maximize
throughput depends upon the application and network environment. The
various load balancing modes each have strengths and weaknesses in
different environments, as detailed below.
For this discussion, we will break down the topologies into
two categories. Depending upon the destination of most traffic, we
categorize them into either "gatewayed" or "local" configurations.
In a gatewayed configuration, the "switch" is acting primarily
as a router, and the majority of traffic passes through this router to
other networks. An example would be the following::
+----------+ +----------+
| |eth0 port1| | to other networks
| Host A +---------------------+ router +------------------->
| +---------------------+ | Hosts B and C are out
| |eth1 port2| | here somewhere
+----------+ +----------+
The router may be a dedicated router device, or another host
acting as a gateway. For our discussion, the important point is that
the majority of traffic from Host A will pass through the router to
some other network before reaching its final destination.
In a gatewayed network configuration, although Host A may
communicate with many other systems, all of its traffic will be sent
and received via one other peer on the local network, the router.
Note that the case of two systems connected directly via
multiple physical links is, for purposes of configuring bonding, the
same as a gatewayed configuration. In that case, it happens that all
traffic is destined for the "gateway" itself, not some other network
beyond the gateway.
In a local configuration, the "switch" is acting primarily as
a switch, and the majority of traffic passes through this switch to
reach other stations on the same network. An example would be the
following::
+----------+ +----------+ +--------+
| |eth0 port1| +-------+ Host B |
| Host A +------------+ switch |port3 +--------+
| +------------+ | +--------+
| |eth1 port2| +------------------+ Host C |
+----------+ +----------+port4 +--------+
Again, the switch may be a dedicated switch device, or another
host acting as a gateway. For our discussion, the important point is
that the majority of traffic from Host A is destined for other hosts
on the same local network (Hosts B and C in the above example).
In summary, in a gatewayed configuration, traffic to and from
the bonded device will be to the same MAC level peer on the network
(the gateway itself, i.e., the router), regardless of its final
destination. In a local configuration, traffic flows directly to and
from the final destinations, thus, each destination (Host B, Host C)
will be addressed directly by their individual MAC addresses.
This distinction between a gatewayed and a local network
configuration is important because many of the load balancing modes
available use the MAC addresses of the local network source and
destination to make load balancing decisions. The behavior of each
mode is described below.
Single-switch 처리량 mode 선택
2382-2523`balance-rr`은 한 TCP/IP connection packet을 여러 interface에 stripe할 수 있는 유일한 mode라 한 stream이 interface 하나 이상의 bandwidth를 쓸 수 있습니다. 하지만 packet reorder로 TCP congestion control과 retransmission을 유발합니다. `net.ipv4.tcp_reordering`을 조절할 수 있으나 TCP도 감지 시 자동 증가합니다. Reorder 비율은 NIC·switch·topology에 따라 달라지고 빠른 card·many-to-many에서 커집니다.
Switch가 IP·MAC으로 port를 고정하면 한 connection은 한 interface bandwidth만 씁니다. UDP처럼 out-of-order를 허용하는 application은 interface 증가에 가까운 선형 single-stream 성능을 얻을 수 있습니다. Switch port는 EtherChannel·trunk로 묶어야 합니다.
`active-backup`은 같은 peer에 backup이 붙은 topology에서 bandwidth 이점이 없지만 switch가 load-balance mode를 지원하지 않아도 구성 없이 사용할 수 있습니다.
`balance-xor`은 특정 peer traffic을 항상 같은 interface로 보내므로 destination MAC이 다양한 local network에 적합하고 모든 traffic이 router 하나로 가는 gatewayed 환경에는 부적합할 수 있습니다. Switch trunk 구성이 필요합니다.
`broadcast`도 single-switch 처리량 관점에서 이점이 거의 없습니다.
`802.3ad`는 표준이라 peer 간 호환성과 aggregate 자동 구성이 좋고 frame 순서를 보장해 한 connection reorder를 피합니다. 모든 device가 같은 speed·duplex여야 하고 balance-rr 외 mode처럼 한 connection은 한 interface bandwidth만 씁니다. Linux 기본 MAC XOR은 gatewayed outgoing traffic을 한 device에 몰 수 있고 incoming은 peer policy에 달렸지만 local 환경은 분산됩니다. MII monitor만 사용합니다.
`balance-tlb`는 MAC peer별 outgoing을 intelligent하게 분산해 local 환경에 좋지만 gatewayed에서는 한 device에 몰립니다. 서로 다른 speed를 허용하고 switch 구성은 필요 없지만 inbound는 한 interface이며 base driver ethtool 지원이 필요하고 ARP monitor를 쓸 수 없습니다.
`balance-alb`는 TLB의 제약·기능에 local peer inbound balancing을 더합니다. 추가로 열린 device의 hardware address 변경을 driver가 지원해야 합니다.
12.1.2 Link monitor는 mode에 따라 제한됩니다. Advanced load balancing mode는 ARP monitor를 지원하지 않아 end-to-end assurance가 더 낮은 MII monitor를 써야 합니다.
12.1.1 MT Bonding Mode Selection for Single Switch Topology
-----------------------------------------------------------
This configuration is the easiest to set up and to understand,
although you will have to decide which bonding mode best suits your
needs. The trade offs for each mode are detailed below:
balance-rr:
This mode is the only mode that will permit a single
TCP/IP connection to stripe traffic across multiple
interfaces. It is therefore the only mode that will allow a
single TCP/IP stream to utilize more than one interface's
worth of throughput. This comes at a cost, however: the
striping generally results in peer systems receiving packets out
of order, causing TCP/IP's congestion control system to kick
in, often by retransmitting segments.
It is possible to adjust TCP/IP's congestion limits by
altering the net.ipv4.tcp_reordering sysctl parameter. The
usual default value is 3. But keep in mind TCP stack is able
to automatically increase this when it detects reorders.
Note that the fraction of packets that will be delivered out of
order is highly variable, and is unlikely to be zero. The level
of reordering depends upon a variety of factors, including the
networking interfaces, the switch, and the topology of the
configuration. Speaking in general terms, higher speed network
cards produce more reordering (due to factors such as packet
coalescing), and a "many to many" topology will reorder at a
higher rate than a "many slow to one fast" configuration.
Many switches do not support any modes that stripe traffic
(instead choosing a port based upon IP or MAC level addresses);
for those devices, traffic for a particular connection flowing
through the switch to a balance-rr bond will not utilize greater
than one interface's worth of bandwidth.
If you are utilizing protocols other than TCP/IP, UDP for
example, and your application can tolerate out of order
delivery, then this mode can allow for single stream datagram
performance that scales near linearly as interfaces are added
to the bond.
This mode requires the switch to have the appropriate ports
configured for "etherchannel" or "trunking."
active-backup:
There is not much advantage in this network topology to
the active-backup mode, as the inactive backup devices are all
connected to the same peer as the primary. In this case, a
load balancing mode (with link monitoring) will provide the
same level of network availability, but with increased
available bandwidth. On the plus side, active-backup mode
does not require any configuration of the switch, so it may
have value if the hardware available does not support any of
the load balance modes.
balance-xor:
This mode will limit traffic such that packets destined
for specific peers will always be sent over the same
interface. Since the destination is determined by the MAC
addresses involved, this mode works best in a "local" network
configuration (as described above), with destinations all on
the same local network. This mode is likely to be suboptimal
if all your traffic is passed through a single router (i.e., a
"gatewayed" network configuration, as described above).
As with balance-rr, the switch ports need to be configured for
"etherchannel" or "trunking."
broadcast:
Like active-backup, there is not much advantage to this
mode in this type of network topology.
802.3ad:
This mode can be a good choice for this type of network
topology. The 802.3ad mode is an IEEE standard, so all peers
that implement 802.3ad should interoperate well. The 802.3ad
protocol includes automatic configuration of the aggregates,
so minimal manual configuration of the switch is needed
(typically only to designate that some set of devices is
available for 802.3ad). The 802.3ad standard also mandates
that frames be delivered in order (within certain limits), so
in general single connections will not see misordering of
packets. The 802.3ad mode does have some drawbacks: the
standard mandates that all devices in the aggregate operate at
the same speed and duplex. Also, as with all bonding load
balance modes other than balance-rr, no single connection will
be able to utilize more than a single interface's worth of
bandwidth.
Additionally, the linux bonding 802.3ad implementation
distributes traffic by peer (using an XOR of MAC addresses
and packet type ID), so in a "gatewayed" configuration, all
outgoing traffic will generally use the same device. Incoming
traffic may also end up on a single device, but that is
dependent upon the balancing policy of the peer's 802.3ad
implementation. In a "local" configuration, traffic will be
distributed across the devices in the bond.
Finally, the 802.3ad mode mandates the use of the MII monitor,
therefore, the ARP monitor is not available in this mode.
balance-tlb:
The balance-tlb mode balances outgoing traffic by peer.
Since the balancing is done according to MAC address, in a
"gatewayed" configuration (as described above), this mode will
send all traffic across a single device. However, in a
"local" network configuration, this mode balances multiple
local network peers across devices in a vaguely intelligent
manner (not a simple XOR as in balance-xor or 802.3ad mode),
so that mathematically unlucky MAC addresses (i.e., ones that
XOR to the same value) will not all "bunch up" on a single
interface.
Unlike 802.3ad, interfaces may be of differing speeds, and no
special switch configuration is required. On the down side,
in this mode all incoming traffic arrives over a single
interface, this mode requires certain ethtool support in the
network device driver of the slave interfaces, and the ARP
monitor is not available.
balance-alb:
This mode is everything that balance-tlb is, and more.
It has all of the features (and restrictions) of balance-tlb,
and will also balance incoming traffic from local network
peers (as described in the Bonding Module Options section,
above).
The only additional down side to this mode is that the network
device driver must support changing the hardware address while
the device is open.
12.1.2 MT Link Monitoring for Single Switch Topology
----------------------------------------------------
The choice of link monitoring may largely depend upon which
mode you choose to use. The more advanced load balancing modes do not
support the use of the ARP monitor, and are thus restricted to using
the MII monitor (which does not provide as high a level of end to end
assurance as the ARP monitor).
병렬 multi-switch 최대 처리량
2524-258112.2 여러 switch를 서로 격리해 Host A와 Host B가 각각 switch A·B·C에 병렬 link를 연결하는 고성능 cluster topology를 만들 수 있습니다. 24 host에서 큰 72-port switch 하나보다 작은 24-port switch 세 개가 저렴할 수 있습니다. 외부 접근은 한 host에 별도 NIC를 두어 gateway로 만듭니다.
12.2.1 실제로는 balance-rr을 주로 사용합니다. Packet coalescing을 하지 않는 adapter를 쓰면 reorder 문제가 완화되고 한 host 간 connection이 여러 interface bandwidth를 활용할 수 있습니다.
12.2.2 Availability보다 성능을 우선해 보통 MII monitor를 씁니다. ARP도 작동하지만 모든 host가 bonding을 쓰는 큰 network에서는 필요한 probe 수가 늘어 이점이 줄어듭니다.
12.2 Maximum Throughput in a Multiple Switch Topology
-----------------------------------------------------
Multiple switches may be utilized to optimize for throughput
when they are configured in parallel as part of an isolated network
between two or more systems, for example::
+-----------+
| Host A |
+-+---+---+-+
| | |
+--------+ | +---------+
| | |
+------+---+ +-----+----+ +-----+----+
| Switch A | | Switch B | | Switch C |
+------+---+ +-----+----+ +-----+----+
| | |
+--------+ | +---------+
| | |
+-+---+---+-+
| Host B |
+-----------+
In this configuration, the switches are isolated from one
another. One reason to employ a topology such as this is for an
isolated network with many hosts (a cluster configured for high
performance, for example), using multiple smaller switches can be more
cost effective than a single larger switch, e.g., on a network with 24
hosts, three 24 port switches can be significantly less expensive than
a single 72 port switch.
If access beyond the network is required, an individual host
can be equipped with an additional network device connected to an
external network; this host then additionally acts as a gateway.
12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
-------------------------------------------------------------
In actual practice, the bonding mode typically employed in
configurations of this type is balance-rr. Historically, in this
network configuration, the usual caveats about out of order packet
delivery are mitigated by the use of network adapters that do not do
any kind of packet coalescing (via the use of NAPI, or because the
device itself does not generate interrupts until some number of
packets has arrived). When employed in this fashion, the balance-rr
mode allows individual connections between two hosts to effectively
utilize greater than one interface's bandwidth.
12.2.2 MT Link Monitoring for Multiple Switch Topology
------------------------------------------------------
Again, in actual practice, the MII monitor is most often used
in this configuration, as performance is given preference over
availability. The ARP monitor will function in this topology, but its
advantages over the MII monitor are mitigated by the volume of probes
needed as the number of systems involved grows (remember that each
host in the network is configured with bonding).
Switch link delay와 duplicate packet
2582-266213.1 일부 switch는 carrier up을 먼저 알리고 autonegotiation·routing protocol·초기화가 끝날 때까지 traffic을 전달하지 않거나 초기화 중 link state를 여러 번 bounce합니다. `updelay`로 interface 사용을 늦춥니다.
Active link가 하나도 없으면 driver는 downtime을 줄이려고 `updelay`를 무시하고 가장 먼저 up 대기 상태가 된 link를 즉시 재사용합니다. Switch가 backup mode로 전환하는 데 오래 걸리면 `downdelay`로 failover를 늦출 수 있습니다.
13.2 Bonding 3.0.2부터 duplicate suppression logic이 있어 문제가 대부분 사라졌지만 설명을 참고용으로 남깁니다. 첫 사용이나 idle 뒤 ping에서 slave 수만큼 duplicate reply가 잠깐 보일 수 있습니다.
이는 driver error가 아니라 switch MAC forwarding table 학습 과정입니다. 처음에는 source MAC의 port를 몰라 모든 port로 flood하고 bond의 여러 slave가 같은 frame을 각각 받습니다. Switch마다 다르며 MAC table을 지우면 재현할 수 있습니다.
13. Switch Behavior Issues
==========================
13.1 Link Establishment and Failover Delays
-------------------------------------------
Some switches exhibit undesirable behavior with regard to the
timing of link up and down reporting by the switch.
First, when a link comes up, some switches may indicate that
the link is up (carrier available), but not pass traffic over the
interface for some period of time. This delay is typically due to
some type of autonegotiation or routing protocol, but may also occur
during switch initialization (e.g., during recovery after a switch
failure). If you find this to be a problem, specify an appropriate
value to the updelay bonding module option to delay the use of the
relevant interface(s).
Second, some switches may "bounce" the link state one or more
times while a link is changing state. This occurs most commonly while
the switch is initializing. Again, an appropriate updelay value may
help.
Note that when a bonding interface has no active links, the
driver will immediately reuse the first link that goes up, even if the
updelay parameter has been specified (the updelay is ignored in this
case). If there are slave interfaces waiting for the updelay timeout
to expire, the interface that first went into that state will be
immediately reused. This reduces down time of the network if the
value of updelay has been overestimated, and since this occurs only in
cases with no connectivity, there is no additional penalty for
ignoring the updelay.
In addition to the concerns about switch timings, if your
switches take a long time to go into backup mode, it may be desirable
to not activate a backup interface immediately after a link goes down.
Failover may be delayed via the downdelay bonding module option.
13.2 Duplicated Incoming Packets
--------------------------------
NOTE: Starting with version 3.0.2, the bonding driver has logic to
suppress duplicate packets, which should largely eliminate this problem.
The following description is kept for reference.
It is not uncommon to observe a short burst of duplicated
traffic when the bonding device is first used, or after it has been
idle for some period of time. This is most easily observed by issuing
a "ping" to some other host on the network, and noticing that the
output from ping flags duplicates (typically one per slave).
For example, on a bond in active-backup mode with five slaves
all connected to one switch, the output may appear as follows::
# ping -n 10.0.4.2
PING 10.0.4.2 (10.0.4.2) from 10.0.3.10 : 56(84) bytes of data.
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.7 ms
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
64 bytes from 10.0.4.2: icmp_seq=2 ttl=64 time=0.216 ms
64 bytes from 10.0.4.2: icmp_seq=3 ttl=64 time=0.267 ms
64 bytes from 10.0.4.2: icmp_seq=4 ttl=64 time=0.222 ms
This is not due to an error in the bonding driver, rather, it
is a side effect of how many switches update their MAC forwarding
tables. Initially, the switch does not associate the MAC address in
the packet with a particular switch port, and so it may send the
traffic to all ports until its MAC forwarding table is updated. Since
the interfaces attached to the bond may occupy multiple ports on a
single switch, when the switch (temporarily) floods the traffic to all
ports, the bond device receives multiple copies of the same packet
(one per slave device).
The duplicated packet behavior is switch dependent, some
switches exhibit this, and some do not. On switches that display this
behavior, it can be induced by clearing the MAC forwarding table (on
most Cisco switches, the privileged command "clear mac address-table
dynamic" will accomplish this).
IBM BladeCenter 고려사항
2663-277214.1 JS20 계열 BladeCenter에서는 topology 때문에 balance-rr, active-backup, balance-tlb, balance-alb만 지원합니다.
JS20 planar에는 Broadcom Gigabit eth0·eth1이 있고 chassis에서 각각 I/O module 1·2에 고정 연결됩니다. Daughter card의 eth2·eth3은 module 3·4로 갑니다. 각 I/O module은 internal switch 또는 외부 switch에 직접 연결하는 optical/copper passthrough module일 수 있습니다.
보통 module 1·2의 Ethernet Switch Module을 쓰면 eth0·eth1이 서로 다른 internal switch에 연결됩니다. Passthrough module을 쓰면 두 interface를 공통 external switch로 보낼 수 있습니다. 전부 passthrough면 single-switch, ESM이 하나라도 있으면 multi-switch topology로 보이며 ESM끼리 연결할 수도 있습니다.
Balance-rr은 bond link 모두 passthrough를 통해 같은 external switch로 가고 switch port가 EtherChannel·trunk여야 합니다. Balance-alb·tlb는 switch·passthrough 혼합도 가능하지만 모든 interface가 모든 destination에 도달하도록 외부에서 network가 수렴해야 합니다. Active-backup은 추가 요구가 없습니다.
ESM이 있으면 external switch까지의 link loss는 ARP monitor만 확실히 감지합니다. Cabinet 외부 port와 JS20 NIC 사이에 internal switch가 있어 MII는 JS20-ESM 구간만 봅니다. Passthrough면 MII가 external port failure를 직접 감지합니다.
Serial over LAN은 primary eth0만 사용해 eth0 link loss 때 다른 traffic처럼 failover하지 않습니다. Bonding control 밖의 system이기 때문입니다. Failover delay를 줄이려고 internal·external switch의 spanning tree를 끄는 것이 유리할 수 있습니다.
14. Hardware Specific Considerations
====================================
This section contains additional information for configuring
bonding on specific hardware platforms, or for interfacing bonding
with particular switches or other devices.
14.1 IBM BladeCenter
--------------------
This applies to the JS20 and similar systems.
On the JS20 blades, the bonding driver supports only
balance-rr, active-backup, balance-tlb and balance-alb modes. This is
largely due to the network topology inside the BladeCenter, detailed
below.
JS20 network adapter information
--------------------------------
All JS20s come with two Broadcom Gigabit Ethernet ports
integrated on the planar (that's "motherboard" in IBM-speak). In the
BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to
I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2.
An add-on Broadcom daughter card can be installed on a JS20 to provide
two more Gigabit Ethernet ports. These ports, eth2 and eth3, are
wired to I/O Modules 3 and 4, respectively.
Each I/O Module may contain either a switch or a passthrough
module (which allows ports to be directly connected to an external
switch). Some bonding modes require a specific BladeCenter internal
network topology in order to function; these are detailed below.
Additional BladeCenter-specific networking information can be
found in two IBM Redbooks (www.ibm.com/redbooks):
- "IBM eServer BladeCenter Networking Options"
- "IBM eServer BladeCenter Layer 2-7 Network Switching"
BladeCenter networking configuration
------------------------------------
Because a BladeCenter can be configured in a very large number
of ways, this discussion will be confined to describing basic
configurations.
Normally, Ethernet Switch Modules (ESMs) are used in I/O
modules 1 and 2. In this configuration, the eth0 and eth1 ports of a
JS20 will be connected to different internal switches (in the
respective I/O modules).
A passthrough module (OPM or CPM, optical or copper,
passthrough module) connects the I/O module directly to an external
switch. By using PMs in I/O module #1 and #2, the eth0 and eth1
interfaces of a JS20 can be redirected to the outside world and
connected to a common external switch.
Depending upon the mix of ESMs and PMs, the network will
appear to bonding as either a single switch topology (all PMs) or as a
multiple switch topology (one or more ESMs, zero or more PMs). It is
also possible to connect ESMs together, resulting in a configuration
much like the example in "High Availability in a Multiple Switch
Topology," above.
Requirements for specific modes
-------------------------------
The balance-rr mode requires the use of passthrough modules
for devices in the bond, all connected to an common external switch.
That switch must be configured for "etherchannel" or "trunking" on the
appropriate ports, as is usual for balance-rr.
The balance-alb and balance-tlb modes will function with
either switch modules or passthrough modules (or a mix). The only
specific requirement for these modes is that all network interfaces
must be able to reach all destinations for traffic sent over the
bonding device (i.e., the network must converge at some point outside
the BladeCenter).
The active-backup mode has no additional requirements.
Link monitoring issues
----------------------
When an Ethernet Switch Module is in place, only the ARP
monitor will reliably detect link loss to an external switch. This is
nothing unusual, but examination of the BladeCenter cabinet would
suggest that the "external" network ports are the ethernet ports for
the system, when it fact there is a switch between these "external"
ports and the devices on the JS20 system itself. The MII monitor is
only able to detect link failures between the ESM and the JS20 system.
When a passthrough module is in place, the MII monitor does
detect failures to the "external" port, which is then directly
connected to the JS20 system.
Other concerns
--------------
The Serial Over LAN (SoL) link is established over the primary
ethernet (eth0) only, therefore, any loss of link to eth0 will result
in losing your SoL connection. It will not fail over with other
network traffic, as the SoL system is beyond the control of the
bonding driver.
It may be desirable to disable spanning tree on the switch
(either the internal Ethernet Switch Module, or an external switch) to
avoid fail-over delay issues when using bonding.
FAQ
2773-28981. SMP safe인가? 예. 옛 2.0.xx channel bonding patch는 아니었지만 새 driver는 처음부터 SMP safe로 설계했습니다.
2. 어떤 card가 작동하는가? Ethernet card라면 종류·제조사를 섞을 수 있고 대부분 mode에서 speed가 같을 필요도 없습니다. 3.2.1부터 active-backup에서 Infiniband slave도 지원합니다.
3. Bond device 수에는 제한이 없습니다.
4. Bond 하나의 slave 수는 Linux가 지원하는 network interface 수와 system에 설치 가능한 card 수로만 제한됩니다.
5. Slave link가 죽으면 monitoring이 켜진 경우 disable합니다. Active-backup은 backup으로 failover하고 다른 mode는 failed link를 무시합니다. 복구되면 mode에 맞게 다시 bond에 참여합니다. `miimon`은 local carrier, `arp_interval`은 local network의 다른 host까지 연결성을 봅니다. Monitoring이 없으면 failure를 감지하지 못해 packet loss와 성능 저하가 생깁니다.
6. HA에 사용할 수 있으며 관련 절의 topology와 mode 선택을 따릅니다.
7. Balance-rr·xor은 EtherChannel·trunk를 지원하는 system, TLB·ALB는 특별한 switch 없이 필요한 base driver 기능, 802.3ad는 dynamic link aggregation 지원 switch, active-backup은 모든 Layer 2 switch에서 작동합니다.
8. Fixed MAC slave 또는 `fail_over_mac`이면 bond MAC은 active slave MAC입니다. 그 외에는 명시하지 않으면 첫 slave MAC을 bond와 이후 slave에 복사하고 첫 slave를 제거해도 bond를 down·재구성할 때까지 유지합니다. `ifconfig bond0 hw ether ...` 또는 `ip link set bond0 address ...`로 바꿀 수 있습니다. Slave를 detach하면 enslave 전 MAC을 복원합니다.
9. Native XDP 지원 mode는 balance-rr(0), active-backup(1), balance-xor(2), 802.3ad(4)입니다. `vlan+srcmac` hash는 native XDP를 지원하지 않고 다른 mode는 generic mode로 XDP program을 load해야 합니다.
15. Frequently Asked Questions
==============================
1. Is it SMP safe?
-------------------
Yes. The old 2.0.xx channel bonding patch was not SMP safe.
The new driver was designed to be SMP safe from the start.
2. What type of cards will work with it?
-----------------------------------------
Any Ethernet type cards (you can even mix cards - a Intel
EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes,
devices need not be of the same speed.
Starting with version 3.2.1, bonding also supports Infiniband
slaves in active-backup mode.
3. How many bonding devices can I have?
----------------------------------------
There is no limit.
4. How many slaves can a bonding device have?
----------------------------------------------
This is limited only by the number of network interfaces Linux
supports and/or the number of network cards you can place in your
system.
5. What happens when a slave link dies?
----------------------------------------
If link monitoring is enabled, then the failing device will be
disabled. The active-backup mode will fail over to a backup link, and
other modes will ignore the failed link. The link will continue to be
monitored, and should it recover, it will rejoin the bond (in whatever
manner is appropriate for the mode). See the sections on High
Availability and the documentation for each mode for additional
information.
Link monitoring can be enabled via either the miimon or
arp_interval parameters (described in the module parameters section,
above). In general, miimon monitors the carrier state as sensed by
the underlying network device, and the arp monitor (arp_interval)
monitors connectivity to another host on the local network.
If no link monitoring is configured, the bonding driver will
be unable to detect link failures, and will assume that all links are
always available. This will likely result in lost packets, and a
resulting degradation of performance. The precise performance loss
depends upon the bonding mode and network configuration.
6. Can bonding be used for High Availability?
----------------------------------------------
Yes. See the section on High Availability for details.
7. Which switches/systems does it work with?
---------------------------------------------
The full answer to this depends upon the desired mode.
In the basic balance modes (balance-rr and balance-xor), it
works with any system that supports etherchannel (also called
trunking). Most managed switches currently available have such
support, and many unmanaged switches as well.
The advanced balance modes (balance-tlb and balance-alb) do
not have special switch requirements, but do need device drivers that
support specific features (described in the appropriate section under
module parameters, above).
In 802.3ad mode, it works with systems that support IEEE
802.3ad Dynamic Link Aggregation. Most managed and many unmanaged
switches currently available support 802.3ad.
The active-backup mode should work with any Layer-II switch.
8. Where does a bonding device get its MAC address from?
---------------------------------------------------------
When using slave devices that have fixed MAC addresses, or when
the fail_over_mac option is enabled, the bonding device's MAC address is
the MAC address of the active slave.
For other configurations, if not explicitly configured (with
ifconfig or ip link), the MAC address of the bonding device is taken from
its first slave device. This MAC address is then passed to all following
slaves and remains persistent (even if the first slave is removed) until
the bonding device is brought down or reconfigured.
If you wish to change the MAC address, you can set it with
ifconfig or ip link::
# ifconfig bond0 hw ether 00:11:22:33:44:55
# ip link set bond0 address 66:77:88:99:aa:bb
The MAC address can be also changed by bringing down/up the
device and then changing its slaves (or their order)::
# ifconfig bond0 down ; modprobe -r bonding
# ifconfig bond0 .... up
# ifenslave bond0 eth...
This method will automatically take the address from the next
slave that is added.
To restore your slaves' MAC addresses, you need to detach them
from the bond (``ifenslave -d bond0 eth0``). The bonding driver will
then restore the MAC addresses that the slaves had before they were
enslaved.
9. What bonding modes support native XDP?
------------------------------------------
* balance-rr (0)
* active-backup (1)
* balance-xor (2)
* 802.3ad (4)
Note that the vlan+srcmac hash policy does not support native XDP.
For other bonding modes, the XDP program must be loaded with generic mode.
자료와 mailing list
2899-2917최신 bonding driver는 `http://kernel.org`의 최신 Linux kernel에 있고 최신 문서는 kernel source의 `Documentation/networking/bonding.rst`입니다.
개발 논의는 vger.kernel.org의 Linux network mailing list `[email protected]`에서 이루어집니다. 구독·해지 administrative interface는 `http://vger.kernel.org/vger-lists.html#netdev`입니다.
16. Resources and Links
=======================
The latest version of the bonding driver can be found in the latest
version of the linux kernel, found on http://kernel.org
The latest version of this document can be found in the latest kernel
source (named Documentation/networking/bonding.rst).
Discussions regarding the development of the bonding driver take place
on the main Linux network mailing list, hosted at vger.kernel.org. The list
address is:
[email protected]
The administrative interface (to subscribe or unsubscribe) can
be found at:
http://vger.kernel.org/vger-lists.html#netdev
요약·해설
bonding.rst:1-2917Bonding은 여러 link를 하나의 logical interface로 묶지만 mode마다 switch 요구사항, 순서 보장, 수신 분배, link monitor가 다릅니다. 먼저 topology가 single peer인지 multi-switch인지 정하고, 성능과 availability 중 우선순위를 고른 뒤 mode와 monitor를 선택해야 합니다.
일곱 mode의 핵심 동작과 switch 요구사항을 비교합니다.
MII와 ARP는 동시에 사용할 수 없습니다.
Mode에 맞지 않는 switch 구성은 packet loss나 편향을 만듭니다.
Runtime에 bond와 slave를 추가·제거하는 경로입니다.
Active-backup에서 bond와 slave MAC이 바뀌는 방식을 정합니다.
Peer 수와 tunnel·VLAN 구조에 맞춰 분배 key를 고릅니다.
원문 L2237-2247 ASCII topology를 같은 연결 관계로 재구성했습니다.
원문 L2328-2333의 single-peer 구조입니다.
원문 L2355-2360처럼 destination MAC이 여러 개인 구조입니다.
원문 L2531-2545의 격리된 세 switch 구조입니다.
Availability와 throughput 목표에 따른 출발점을 정리했습니다.
All-zero MAC을 복사하지 않도록 순서를 지킵니다.
같은 L2에서 예측 가능한 LACPDU 식별자 spoof 가능성을 낮춥니다.
Internal switch 유무가 MII failure 감지 범위를 바꿉니다.