IPVS Sysctl

Memory 기준, connection 재사용과 기본 기능

IPVS 변수는 `/proc/sys/net/ipv4/vs/*`에 있습니다. `am_droprate` 기본 10은 `drop_packet=3` always mode의 1/rate 폐기율이고 `amemthresh` 기본 1024 page는 자동 defense를 켤 available memory threshold입니다. memory가 부족하면 해당 strategy 값이 2로, 충분하면 1로 자동 전환됩니다.

`backup_only`는 backup mode 중 director 기능을 꺼 DR/TUN packet loop를 막습니다. `conn_reuse_mode`는 source port가 재사용된 새 connection을 같은 real server에 둘지 reschedule할지 정하는 bitmap입니다. 0은 특별 처리 없음, bit 1은 `expire_nodest_conn`이 켜졌거나 NAT TCP가 TIME_WAIT일 때 안전하게 reschedule, bit 2는 Direct Routing이 마지막으로 보는 FIN_WAIT까지 포함해 busy cluster에 새 real server를 추가할 때 유용합니다.

`conntrack`은 IPVS connection에도 conntrack entry를 유지해 stateful firewall rule을 적용합니다. 필요 없으면 끄는 것이 성능상 유리하고 FTP application module은 설정과 무관하게 entry를 만듭니다. `CONFIG_IP_VS_NFCT`가 필요합니다. `cache_bypass`는 usable cache server가 없고 destination이 non-local unicast일 때 transparent web cache traffic을 원 destination으로 직접 전달합니다.

`debug_level`은 0의 transmission error부터 12 이상의 packet traversal까지 누적 verbosity를 올립니다. `CONFIG_IP_VS_DEBUG` build에서만 쓸 수 있고 높은 level은 낮은 level message를 모두 포함합니다.

.. SPDX-License-Identifier: GPL-2.0

===========
IPvs-sysctl
===========

/proc/sys/net/ipv4/vs/* Variables:
==================================

am_droprate - INTEGER
        default 10

        It sets the always mode drop rate, which is used in the mode 3
        of the drop_rate defense.

amemthresh - INTEGER
        default 1024

        It sets the available memory threshold (in pages), which is
        used in the automatic modes of defense. When there is no
        enough available memory, the respective strategy will be
        enabled and the variable is automatically set to 2, otherwise
        the strategy is disabled and the variable is  set  to 1.

backup_only - BOOLEAN
        - 0 - disabled (default)
        - not 0 - enabled

        If set, disable the director function while the server is
        in backup mode to avoid packet loops for DR/TUN methods.

conn_reuse_mode - INTEGER
        1 - default

        Controls how ipvs will deal with connections that are detected
        port reuse. It is a bitmap, with the values being:

        0: disable any special handling on port reuse. The new
        connection will be delivered to the same real server that was
        servicing the previous connection.

        bit 1: enable rescheduling of new connections when it is safe.
        That is, whenever expire_nodest_conn and for TCP sockets, when
        the connection is in TIME_WAIT state (which is only possible if
        you use NAT mode).

        bit 2: it is bit 1 plus, for TCP connections, when connections
        are in FIN_WAIT state, as this is the last state seen by load
        balancer in Direct Routing mode. This bit helps on adding new
        real servers to a very busy cluster.

conntrack - BOOLEAN
        - 0 - disabled (default)
        - not 0 - enabled

        If set, maintain connection tracking entries for
        connections handled by IPVS.

        This should be enabled if connections handled by IPVS are to be
        also handled by stateful firewall rules. That is, iptables rules
        that make use of connection tracking.  It is a performance
        optimisation to disable this setting otherwise.

        Connections handled by the IPVS FTP application module
        will have connection tracking entries regardless of this setting.

        Only available when IPVS is compiled with CONFIG_IP_VS_NFCT enabled.

cache_bypass - BOOLEAN
        - 0 - disabled (default)
        - not 0 - enabled

        If it is enabled, forward packets to the original destination
        directly when no cache server is available and destination
        address is not local (iph->daddr is RTN_UNICAST). It is mostly
        used in transparent web cache cluster.

debug_level - INTEGER
        - 0          - transmission error messages (default)
        - 1          - non-fatal error messages
        - 2          - configuration
        - 3          - destination trash
        - 4          - drop entry
        - 5          - service lookup
        - 6          - scheduling
        - 7          - connection new/expire, lookup and synchronization
        - 8          - state transition
        - 9          - binding destination, template checks and applications
        - 10         - IPVS packet transmission
        - 11         - IPVS packet handling (ip_vs_in/ip_vs_out)
        - 12 or more - packet traversal

        Only available when IPVS is compiled with CONFIG_IP_VS_DEBUG enabled.

        Higher debugging levels include the messages for lower debugging
        levels, so setting debug level 2, includes level 0, 1 and 2
        messages. Thus, logging becomes more and more verbose the higher
        the level.

Connection table 방어와 estimator

`drop_entry`는 connection hash table의 1/32을 무작위 scan해 SYN-RECV/SYNACK entry를 떨어뜨려 새 connection memory를 회수하는 SYN flood 방어입니다. 0=항상 끔, 1/2=memory에 따라 자동 off/on, 3=항상 켬입니다.

`drop_packet`은 real server로 forwarding하기 전에 1/rate packet을 버립니다. rate 1이면 모두 버립니다. mode 1/2에서는 available memory가 `amemthresh`보다 작을 때 `rate = amemthresh / (amemthresh - available_memory)`로 계산하고 mode 3에서는 `am_droprate`를 씁니다.

`est_cpulist`는 estimation kthread가 실행될 CPU를 표준 cpulist로 제한합니다. 빈 목록은 kthread와 estimation을 멈추고 기본은 system housekeeping CPU입니다. `est_nice`는 -20~19의 scheduler niceness로 기본 0입니다.

drop_entry - INTEGER
        - 0  - disabled (default)

        The drop_entry defense is to randomly drop entries in the
        connection hash table, just in order to collect back some
        memory for new connections. In the current code, the
        drop_entry procedure can be activated every second, then it
        randomly scans 1/32 of the whole and drops entries that are in
        the SYN-RECV/SYNACK state, which should be effective against
        syn-flooding attack.

        The valid values of drop_entry are from 0 to 3, where 0 means
        that this strategy is always disabled, 1 and 2 mean automatic
        modes (when there is no enough available memory, the strategy
        is enabled and the variable is automatically set to 2,
        otherwise the strategy is disabled and the variable is set to
        1), and 3 means that the strategy is always enabled.

drop_packet - INTEGER
        - 0  - disabled (default)

        The drop_packet defense is designed to drop 1/rate packets
        before forwarding them to real servers. If the rate is 1, then
        drop all the incoming packets.

        The value definition is the same as that of the drop_entry. In
        the automatic mode, the rate is determined by the follow
        formula: rate = amemthresh / (amemthresh - available_memory)
        when available memory is less than the available memory
        threshold. When the mode 3 is set, the always mode drop rate
        is controlled by the /proc/sys/net/ipv4/vs/am_droprate.

est_cpulist - CPULIST
        Allowed        CPUs for estimation kthreads

        Syntax: standard cpulist format
        empty list - stop kthread tasks and estimation
        default - the system's housekeeping CPUs for kthreads

        Example:
        "all": all possible CPUs
        "0-N": all possible CPUs, N denotes last CPU number
        "0,1-N:1/2": first and all CPUs with odd number
        "": empty list

est_nice - INTEGER
        default 0
        Valid range: -20 (more favorable) .. 19 (less favorable)

        Niceness value to use for the estimation kthreads (scheduling
        priority)

Destination 소멸, tunnel과 PMTU

`expire_nodest_conn=0`은 destination server가 없어도 packet을 조용히 버려 monitor가 server를 다시 추가하면 기존 connection이 이어질 수 있게 합니다. 1은 packet 도착 즉시 connection을 expire해 client에 종료를 알립니다. `expire_quiescent_template`은 weight 0인 quiescent server를 가리키는 persistence template을 새 scheduling 시 expire해 다음 persistent connection이 다른 server로 가게 합니다.

`ignore_tunneled`은 인식하지 못한 protocol packet에 `ipvs_property`를 표시해 이미 IPIP 같은 tunnel로 IPVS host에 온 packet을 다시 schedule하는 routing loop를 막습니다. `nat_icmp_send`는 VS/NAT에서 real server발 packet에 해당 connection entry가 없을 때 `ICMP_DEST_UNREACH`를 보냅니다.

`pmtu_disc` 기본 1은 forwarding method와 무관하게 PMTU를 넘는 DF packet을 FRAG_NEEDED로 거부합니다. TUN method에서 0으로 두면 그런 packet을 fragment할 수 있습니다. `secure_tcp`는 더 엄격한 TCP state transition table을 쓰고 VS/NAT에서 3-way handshake 완료까지 ESTABLISHED 진입을 늦춥니다. 값 0~3의 의미는 drop defense와 같습니다.

expire_nodest_conn - BOOLEAN
        - 0 - disabled (default)
        - not 0 - enabled

        The default value is 0, the load balancer will silently drop
        packets when its destination server is not available. It may
        be useful, when user-space monitoring program deletes the
        destination server (because of server overload or wrong
        detection) and add back the server later, and the connections
        to the server can continue.

        If this feature is enabled, the load balancer will expire the
        connection immediately when a packet arrives and its
        destination server is not available, then the client program
        will be notified that the connection is closed. This is
        equivalent to the feature some people requires to flush
        connections when its destination is not available.

expire_quiescent_template - BOOLEAN
        - 0 - disabled (default)
        - not 0 - enabled

        When set to a non-zero value, the load balancer will expire
        persistent templates when the destination server is quiescent.
        This may be useful, when a user makes a destination server
        quiescent by setting its weight to 0 and it is desired that
        subsequent otherwise persistent connections are sent to a
        different destination server.  By default new persistent
        connections are allowed to quiescent destination servers.

        If this feature is enabled, the load balancer will expire the
        persistence template if it is to be used to schedule a new
        connection and the destination server is quiescent.

ignore_tunneled - BOOLEAN
        - 0 - disabled (default)
        - not 0 - enabled

        If set, ipvs will set the ipvs_property on all packets which are of
        unrecognized protocols.  This prevents us from routing tunneled
        protocols like ipip, which is useful to prevent rescheduling
        packets that have been tunneled to the ipvs host (i.e. to prevent
        ipvs routing loops when ipvs is also acting as a real server).

nat_icmp_send - BOOLEAN
        - 0 - disabled (default)
        - not 0 - enabled

        It controls sending icmp error messages (ICMP_DEST_UNREACH)
        for VS/NAT when the load balancer receives packets from real
        servers but the connection entries don't exist.

pmtu_disc - BOOLEAN
        - 0 - disabled
        - not 0 - enabled (default)

        By default, reject with FRAG_NEEDED all DF packets that exceed
        the PMTU, irrespective of the forwarding method. For TUN method
        the flag can be disabled to fragment such packets.

secure_tcp - INTEGER
        - 0  - disabled (default)

        The secure_tcp defense is to use a more complicated TCP state
        transition table. For VS/NAT, it also delays entering the
        TCP ESTABLISHED state until the three way handshake is completed.

        The value definition is the same as that of drop_entry and
        drop_packet.

Connection synchronization과 SNAT route

`sync_threshold` 기본 `3 50`은 connection이 최소 몇 incoming packet 뒤 처음 sync되고 이후 packet count modulo `sync_period`가 threshold일 때 다시 sync될지 정합니다. period와 `sync_refresh_period`가 모두 0이면 state change 또는 threshold 첫 일치 때만 보냅니다.

`sync_refresh_period`은 마지막 보고 timer와 새 timer의 차이가 sync를 유발할 초 단위 기준입니다. state가 안 바뀐 high-traffic connection의 sync rate를 줄이며 그 1/8 간격으로 `sync_retries` 0~3회 재시도할 수 있습니다. `sync_qlen_max`는 아직 보내지 못한 sync message queue의 hard limit으로 기본 memory page 수의 1/32이지만 단위는 message 수입니다. `sync_sock_size=0`은 master SNDBUF와 slave RCVBUF에 system default를 유지합니다.

`sync_ports`는 master/backup sync thread와 UDP port 수입니다. thread 0은 8848, 마지막은 `8848 + sync_ports - 1`을 씁니다. `snat_reroute`는 real server에서 온 SNAT packet을 director가 origin인 것처럼 route lookup해 policy routing 차이를 반영합니다. policy routing이 없다면 결과가 같으므로 0으로 route 재계산 비용을 줄일 수 있습니다.

sync_threshold - vector of 2 INTEGERs: sync_threshold, sync_period
        default 3 50

        It sets synchronization threshold, which is the minimum number
        of incoming packets that a connection needs to receive before
        the connection will be synchronized. A connection will be
        synchronized, every time the number of its incoming packets
        modulus sync_period equals the threshold. The range of the
        threshold is from 0 to sync_period.

        When sync_period and sync_refresh_period are 0, send sync only
        for state changes or only once when pkts matches sync_threshold

sync_refresh_period - UNSIGNED INTEGER
        default 0

        In seconds, difference in reported connection timer that triggers
        new sync message. It can be used to avoid sync messages for the
        specified period (or half of the connection timeout if it is lower)
        if connection state is not changed since last sync.

        This is useful for normal connections with high traffic to reduce
        sync rate. Additionally, retry sync_retries times with period of
        sync_refresh_period/8.

sync_retries - INTEGER
        default 0

        Defines sync retries with period of sync_refresh_period/8. Useful
        to protect against loss of sync messages. The range of the
        sync_retries is from 0 to 3.

sync_qlen_max - UNSIGNED LONG

        Hard limit for queued sync messages that are not sent yet. It
        defaults to 1/32 of the memory pages but actually represents
        number of messages. It will protect us from allocating large
        parts of memory when the sending rate is lower than the queuing
        rate.

sync_sock_size - INTEGER
        default 0

        Configuration of SNDBUF (master) or RCVBUF (slave) socket limit.
        Default value is 0 (preserve system defaults).

sync_ports - INTEGER
        default 1

        The number of threads that master and backup servers can use for
        sync traffic. Every thread will use single UDP port, thread 0 will
        use the default port 8848 while last thread will use port
        8848+sync_ports-1.

snat_reroute - BOOLEAN
        - 0 - disabled
        - not 0 - enabled (default)

        If enabled, recalculate the route of SNATed packets from
        realservers so that they are routed as if they originate from the
        director. Otherwise they are routed as if they are forwarded by the
        director.

        If policy routing is in effect then it is possible that the route
        of a packet originating from a director is routed differently to a
        packet being forwarded by the director.

        If policy routing is not in effect then the recalculated route will
        always be the same as the original route so it is an optimisation
        to disable snat_reroute and avoid the recalculation.

Persistence 최적화, protocol version과 estimation

`sync_persist_mode=0`은 모든 connection을 sync합니다. 1은 persistent service에서 normal connection은 생략하고 persistence template만 sync해 traffic을 줄입니다. backup의 TCP/SCTP에는 `sloppy_tcp`, `sloppy_sctp`가 필요할 수 있고 non-persistent service에는 mode 0으로 동작합니다.

`sync_version=0`은 legacy original protocol, 1은 권장되는 current protocol을 송신합니다. 이 entry가 있는 kernel은 두 protocol version의 message를 수신할 수 있습니다. `run_estimation=0`은 estimator와 kthread를 멈추며 1로 다시 켤 수 있지만 재활성화 직후 첫 estimate는 정확하지 않습니다.

sync_persist_mode - INTEGER
        default 0

        Controls the synchronisation of connections when using persistence

        0: All types of connections are synchronised

        1: Attempt to reduce the synchronisation traffic depending on
        the connection type. For persistent services avoid synchronisation
        for normal connections, do it only for persistence templates.
        In such case, for TCP and SCTP it may need enabling sloppy_tcp and
        sloppy_sctp flags on backup servers. For non-persistent services
        such optimization is not applied, mode 0 is assumed.

sync_version - INTEGER
        default 1

        The version of the synchronisation protocol used when sending
        synchronisation messages.

        0 selects the original synchronisation protocol (version 0). This
        should be used when sending synchronisation messages to a legacy
        system that only understands the original synchronisation protocol.

        1 selects the current synchronisation protocol (version 1). This
        should be used where possible.

        Kernels with this sync_version entry are able to receive messages
        of both version 1 and version 2 of the synchronisation protocol.

run_estimation - BOOLEAN
        0 - disabled
        not 0 - enabled (default)

        If disabled, the estimation will be suspended and kthread tasks
        stopped.

        You can always re-enable estimation by setting this value to 1.
        But be careful, the first estimation after re-enable is not
        accurate.