네트워크 SNMP counter 해설

문서 목적

이 문서는 Linux IP, ICMP, TCP 경로에서 제공하는 SNMP counter의 정확한 의미와 증가 시점을 설명합니다. 단순 이름 풀이가 아니라 GRO/GSO, 오류 처리 순서, TCP 상태와 재전송 알고리즘 때문에 counter들이 어떻게 달라지는지를 다룹니다.

============
SNMP counter
============

This document explains the meaning of SNMP counters.

일반 IPv4 counter

일반 IPv4 counter는 모든 L4 packet과 ICMP packet의 영향을 받지만 STP 같은 L2 packet이나 ARP에는 변하지 않습니다. `IpInReceives`는 `ip_rcv()` 시작에서 증가하며 항상 `IpExtInOctets`와 함께 갱신됩니다. 이후 잘못된 IP header나 checksum 때문에 drop되어도 이미 증가합니다. GRO/LRO 뒤의 aggregated segment 수를 세므로 실제 wire frame 수보다 작을 수 있습니다.

`IpInDelivers`는 유효 IP packet을 TCP, UDP, ICMP 같은 상위 protocol에 전달한 수입니다. raw socket listener가 없으면 kernel 지원 protocol만 전달하지만 raw socket이 있으면 모든 유효 IP packet이 전달됩니다. `IpOutRequests`는 unicast와 multicast를 포함해 IP 계층으로 송신한 packet 수이며 `IpExtOutOctets`와 함께 증가합니다.

`IpExtInOctets`와 `IpExtOutOctets`는 Linux 확장입니다. RFC1213의 `ifInOctets`·`ifOutOctets`와 달리 MAC header를 포함하지 않고 IP header와 IP data만 셉니다. ECN별 `IpExtInNoECTPkts`, `IpExtInECT1Pkts`, `IpExtInECT0Pkts`, `IpExtInCEPkts`는 LRO/GRO와 무관한 실제 frame 수를 상태별로 세므로 같은 트래픽에서 `IpInReceives=1`인데 ECN counter가 2 이상일 수 있습니다.

`IpInHdrErrors`는 input 또는 forward 경로에서 IP header 오류로 drop한 수입니다. `IpInAddrErrors`는 IP address 자체가 유효하지 않거나 destination이 local이 아닌데 forwarding이 꺼진 경우 증가합니다. forwarding이 켜졌지만 route table에서 목적지 route를 찾지 못해 수신 packet을 버리면 `IpExtInNoRoutes`가 증가합니다.

`IpInUnknownProtos`는 kernel이 지원하지 않는 L4 protocol일 때 증가하지만 raw socket이 받으면 증가하지 않습니다. `IpExtInTruncatedPkts`는 실제 IPv4 data 크기가 header의 Total Length보다 작은 경우입니다. `IpInDiscards`와 `IpOutDiscards`는 각각 수신·송신 IP 경로에서 메모리 부족 같은 kernel 내부 사유로 버린 packet을 뜻합니다. 송신할 route가 없어 버리면 `IpOutNoRoutes`입니다.

General IPv4 counters
=====================
All layer 4 packets and ICMP packets will change these counters, but
these counters won't be changed by layer 2 packets (such as STP) or
ARP packets.

* IpInReceives

Defined in `RFC1213 ipInReceives`_

.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26

The number of packets received by the IP layer. It gets increasing at the
beginning of ip_rcv function, always be updated together with
IpExtInOctets. It will be increased even if the packet is dropped
later (e.g. due to the IP header is invalid or the checksum is wrong
and so on).  It indicates the number of aggregated segments after
GRO/LRO.

* IpInDelivers

Defined in `RFC1213 ipInDelivers`_

.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28

The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
ICMP and so on. If no one listens on a raw socket, only kernel
supported protocols will be delivered, if someone listens on the raw
socket, all valid IP packets will be delivered.

* IpOutRequests

Defined in `RFC1213 ipOutRequests`_

.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28

The number of packets sent via IP layer, for both single cast and
multicast packets, and would always be updated together with
IpExtOutOctets.

* IpExtInOctets and IpExtOutOctets

They are Linux kernel extensions, no RFC definitions. Please note,
RFC1213 indeed defines ifInOctets  and ifOutOctets, but they
are different things. The ifInOctets and ifOutOctets include the MAC
layer header size but IpExtInOctets and IpExtOutOctets don't, they
only include the IP layer header and the IP layer data.

* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts

They indicate the number of four kinds of ECN IP packets, please refer
`Explicit Congestion Notification`_ for more details.

.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6

These 4 counters calculate how many packets received per ECN
status. They count the real frame number regardless the LRO/GRO. So
for the same packet, you might find that IpInReceives count 1, but
IpExtInNoECTPkts counts 2 or more.

* IpInHdrErrors

Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
dropped due to the IP header error. It might happen in both IP input
and IP forward paths.

.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27

* IpInAddrErrors

Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
scenarios: (1) The IP address is invalid. (2) The destination IP
address is not a local address and IP forwarding is not enabled

.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27

* IpExtInNoRoutes

This counter means the packet is dropped when the IP stack receives a
packet and can't find a route for it from the route table. It might
happen when IP forwarding is enabled and the destination IP address is
not a local address and there is no route for the destination IP
address.

* IpInUnknownProtos

Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
layer 4 protocol is unsupported by kernel. If an application is using
raw socket, kernel will always deliver the packet to the raw socket
and this counter won't be increased.

.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27

* IpExtInTruncatedPkts

For IPv4 packet, it means the actual data size is smaller than the
"Total Length" field in the IPv4 header.

* IpInDiscards

Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
in the IP receiving path and due to kernel internal reasons (e.g. no
enough memory).

.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28

* IpOutDiscards

Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
dropped in the IP sending path and due to kernel internal reasons.

.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28

* IpOutNoRoutes

Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
dropped in the IP sending path and no route is found for it.

.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29

ICMP counter와 갱신 관계

`IcmpInMsgs`와 `IcmpOutMsgs`는 RFC1213 정의대로 오류 packet도 포함합니다. 잘못된 ICMP type이어도 증가합니다. 출력 경로가 raw socket header를 검사하므로 userspace가 IP header를 구성한 packet도 `IcmpOutMsgs`에 포함됩니다.

이름이 붙은 counter는 Destination Unreachable, Time Exceeded, Parameter Problem, Source Quench, Redirect, Echo, Echo Reply, Timestamp, Timestamp Reply, Address Mask와 Address Mask Reply 각각에 대해 In·Out 쌍을 제공합니다. In은 kernel이 해당 type을 수신한 수, Out은 kernel이 송신한 수입니다.

모든 ICMP type은 숫자형 `IcmpMsgInType[N]`과 `IcmpMsgOutType[N]`으로도 집계합니다. N은 IANA ICMP type number입니다. Echo 송신은 `IcmpMsgOutType8`, Echo Reply 수신은 `IcmpMsgInType0`을 1 증가시킵니다.

`IcmpInCsumErrors`는 잘못된 ICMP checksum입니다. kernel은 먼저 `IcmpInMsgs`를 증가시키고 checksum을 검증한 뒤 숫자형 type counter를 갱신하므로 checksum 오류 packet은 `IcmpInMsgs`에는 들어가지만 어떤 `IcmpMsgInType[N]`에도 들어가지 않습니다. 이때 `IcmpInErrors`도 항상 함께 증가합니다. 수신·송신 handler의 다른 오류는 각각 `IcmpInErrors`, `IcmpOutErrors`에 기록됩니다.

출력에서는 `IcmpMsgOutType[N]` 합계와 `IcmpOutMsgs`가 동시에 갱신되어 항상 같습니다. 입력에서는 먼저 `IcmpInMsgs`, 초기 검증 오류면 `IcmpInErrors`, 정상 type이면 type counter, 이후 type handler 오류면 다시 `IcmpInErrors` 순서입니다. 따라서 오류 시점에 따라 type 합계와 error 합계의 관계가 달라집니다. 원문의 단계 3은 `IcmpMsgOutType[N]`으로 적혀 있으나 수신 절차의 문맥과 앞 설명은 입력 type counter를 가리킵니다. 원문 표기는 아래 원문 블록에 그대로 보존했습니다.

ICMP counters
=============
* IcmpInMsgs and IcmpOutMsgs

Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_

.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43

As mentioned in the RFC1213, these two counters include errors, they
would be increased even if the ICMP packet has an invalid type. The
ICMP output path will check the header of a raw socket, so the
IcmpOutMsgs would still be updated if the IP header is constructed by
a userspace program.

* ICMP named types

| These counters include most of common ICMP types, they are:
| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
| IcmpInRedirects: `RFC1213 icmpInRedirects`_
| IcmpInEchos: `RFC1213 icmpInEchos`_
| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
| IcmpOutEchos: `RFC1213 icmpOutEchos`_
| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_

.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43

.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46

Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
straightforward. The 'In' counter means kernel receives such a packet
and the 'Out' counter means kernel sends such a packet.

* ICMP numeric types

They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
ICMP type number. These counters track all kinds of ICMP packets. The
ICMP type number definition could be found in the `ICMP parameters`_
document.

.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml

For example, if the Linux kernel sends an ICMP Echo packet, the
IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
packet, IcmpMsgInType0 would increase 1.

* IcmpInCsumErrors

This counter indicates the checksum of the ICMP packet is
wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
before updating IcmpMsgInType[N]. If a packet has bad checksum, the
IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.

* IcmpInErrors and IcmpOutErrors

Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_

.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43

When an error occurs in the ICMP packet handler path, these two
counters would be updated. The receiving packet path use IcmpInErrors
and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
is increased, IcmpInErrors would always be increased too.

relationship of the ICMP counters
---------------------------------
The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
are updated at the same time. The sum of IcmpMsgInType[N] plus
IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
receives an ICMP packet, kernel follows below logic:

1. increase IcmpInMsgs
2. if has any error, update IcmpInErrors and finish the process
3. update IcmpMsgOutType[N]
4. handle the packet depending on the type, if has any error, update
   IcmpInErrors and finish the process

So if all errors occur in step (2), IcmpInMsgs should be equal to the
sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
step (4), IcmpInMsgs should be equal to the sum of
IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
IcmpInErrors.

일반 TCP counter

`TcpInSegs`는 TCP 계층이 받은 packet 수이며 checksum이나 TCP header 오류도 포함합니다. 다만 L2 destination이 NIC 주소와 달라 TCP가 초기에 버리는 multicast·broadcast·promiscuous 수신은 증가 전 drop됩니다. GRO를 인식하지 않아 두 packet이 하나로 합쳐지면 1만 증가합니다.

`TcpOutSegs`는 TCP가 보낸 packet 수로 retransmission은 제외하지만 SYN, ACK, RST는 포함합니다. GSO를 인식하므로 하나의 skb가 두 segment로 나뉘면 2 증가합니다. `TcpActiveOpens`는 SYN을 보내 `SYN-SENT`로 들어간 횟수이며 이때 `TcpOutSegs`도 같이 증가합니다. `TcpPassiveOpens`는 SYN을 받고 SYN+ACK를 보내 `SYN-RCVD`가 된 횟수입니다.

응용이 읽기 전 TCP receive queue에서 packet을 합치면 `TcpExtTCPRcvCoalesce`가 합쳐진 packet 수만큼 증가합니다. GRO에서 이미 합쳐진 packet은 여기서 세지 않습니다. 송신 중 작은 packet을 더 큰 packet으로 합치는 auto corking은 합친 packet마다 `TcpExtTCPAutoCorking`을 증가시킵니다.

`TcpExtTCPOrigDataSent`는 retransmission을 제외한 원본 data packet 수이며 SYN 안의 data는 포함하고 pure ACK는 제외합니다. `TcpOutSegs`는 pure ACK까지 포함하므로 retransmission rate를 계산할 때 원본 data 기준인 이 counter가 더 유용합니다. `TCPSynRetrans`는 SYN과 SYN/ACK retransmit을, `TCPFastOpenActiveFail`은 상대가 SYN/data를 받지 않거나 timeout된 Fast Open 시도를 셉니다.

accept queue가 가득 찬 상태에서 SYN을 받으면 kernel 4.10 이상은 SYN을 drop하고 `TcpExtListenOverflows`와 `TcpExtListenDrops`를 모두 증가시킵니다. LISTEN socket에서 다른 사유로 packet을 drop해도 `ListenDrops`는 증가하므로 두 값은 항상 같지 않습니다. 메모리 할당 실패는 overflow 없이 drop만 증가시키는 예입니다. 구형 kernel은 handshake를 끝내고 half-open queue에 유지하며 ACK 때마다 accept queue 이동을 다시 시도했습니다.

General TCP counters
====================
* TcpInSegs

Defined in `RFC1213 tcpInSegs`_

.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48

The number of packets received by the TCP layer. As mentioned in
RFC1213, it includes the packets received in error, such as checksum
error, invalid TCP header and so on. Only one error won't be included:
if the layer 2 destination address is not the NIC's layer 2
address. It might happen if the packet is a multicast or broadcast
packet, or the NIC is in promiscuous mode. In these situations, the
packets would be delivered to the TCP layer, but the TCP layer will discard
these packets before increasing TcpInSegs. The TcpInSegs counter
isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
counter would only increase 1.

* TcpOutSegs

Defined in `RFC1213 tcpOutSegs`_

.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48

The number of packets sent by the TCP layer. As mentioned in RFC1213,
it excludes the retransmitted packets. But it includes the SYN, ACK
and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
increase 2.

* TcpActiveOpens

Defined in `RFC1213 tcpActiveOpens`_

.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47

It means the TCP layer sends a SYN, and come into the SYN-SENT
state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
increase 1.

* TcpPassiveOpens

Defined in `RFC1213 tcpPassiveOpens`_

.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47

It means the TCP layer receives a SYN, replies a SYN+ACK, come into
the SYN-RCVD state.

* TcpExtTCPRcvCoalesce

When packets are received by the TCP layer and are not be read by the
application, the TCP layer will try to merge them. This counter
indicate how many packets are merged in such situation. If GRO is
enabled, lots of packets would be merged by GRO, these packets
wouldn't be counted to TcpExtTCPRcvCoalesce.

* TcpExtTCPAutoCorking

When sending packets, the TCP layer will try to merge small packets to
a bigger one. This counter increase 1 for every packet merged in such
situation. Please refer to the LWN article for more details:
https://lwn.net/Articles/576263/

* TcpExtTCPOrigDataSent

This counter is explained by kernel commit f19c29e3e391, I pasted the
explanation below::

  TCPOrigDataSent: number of outgoing packets with original data (excluding
  retransmission but including data-in-SYN). This counter is different from
  TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
  more useful to track the TCP retransmission rate.

* TCPSynRetrans

This counter is explained by kernel commit f19c29e3e391, I pasted the
explanation below::

  TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
  retransmissions into SYN, fast-retransmits, timeout retransmits, etc.

* TCPFastOpenActiveFail

This counter is explained by kernel commit f19c29e3e391, I pasted the
explanation below::

  TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
  the remote does not accept it or the attempts timed out.

* TcpExtListenOverflows and TcpExtListenDrops

When kernel receives a SYN from a client, and if the TCP accept queue
is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
At the same time kernel will also add 1 to TcpExtListenDrops. When a
TCP socket is in LISTEN state, and kernel need to drop a packet,
kernel would always add 1 to TcpExtListenDrops. So increase
TcpExtListenOverflows would let TcpExtListenDrops increasing at the
same time, but TcpExtListenDrops would also increase without
TcpExtListenOverflows increasing, e.g. a memory allocation fail would
also let TcpExtListenDrops increase.

Note: The above explanation is based on kernel 4.10 or above version, on
an old kernel, the TCP stack has different behavior when TCP accept
queue is full. On the old kernel, TCP stack won't drop the SYN, it
would complete the 3-way handshake. As the accept queue is full, TCP
stack will keep the socket in the TCP half-open queue. As it is in the
half open queue, TCP stack will send SYN+ACK on an exponential backoff
timer, after client replies ACK, TCP stack checks whether the accept
queue is still full, if it is not full, moves the socket to the accept
queue, if it is full, keeps the socket in the half-open queue, at next
time client replies ACK, this socket will get another chance to move
to the accept queue.

RST, keepalive와 spurious timeout

`TcpEstabResets`와 `TcpAttemptFails`는 RFC1213 정의를 따릅니다. Linux의 `TcpOutRsts`는 실제 송신 성공 수가 아니라 RST segment를 보내려고 시도한 수입니다. 메모리 할당 실패 등으로 실제 전송이 실패해도 증가할 수 있습니다.

재전송하려던 packet이 네트워크에서 잃은 것이 아니라 qdisc나 driver queue에 오래 머물러 아직 송신되지 않았음을 발견하면 재전송을 취소하고 `TcpExtTCPSpuriousRtxHostQueues`를 증가시킵니다. Established 또는 CloseWait socket이 RST를 수신하면 `TcpEstabResets`가 증가합니다.

`TcpExtTCPKeepAlive`는 송신한 keepalive packet 수입니다. 기본은 비활성이며 userspace가 `SO_KEEPALIVE`를 설정해야 합니다. `TcpExtTCPSpuriousRTOs`는 F-RTO 알고리즘이 탐지한 가짜 retransmission timeout 수입니다.

TCP Fast Open
=============
* TcpEstabResets

Defined in `RFC1213 tcpEstabResets`_.

.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48

* TcpAttemptFails

Defined in `RFC1213 tcpAttemptFails`_.

.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48

* TcpOutRsts

Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
the 'segments sent containing the RST flag', but in linux kernel, this
counter indicates the segments kernel tried to send. The sending
process might be failed due to some errors (e.g. memory alloc failed).

.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52

* TcpExtTCPSpuriousRtxHostQueues

When the TCP stack wants to retransmit a packet, and finds that packet
is not lost in the network, but the packet is not sent yet, the TCP
stack would give up the retransmission and update this counter. It
might happen if a packet stays too long time in a qdisc or driver
queue.

* TcpEstabResets

The socket receives a RST packet in Establish or CloseWait state.

* TcpExtTCPKeepAlive

This counter indicates many keepalive packets were sent. The keepalive
won't be enabled by default. A userspace program could enable it by
setting the SO_KEEPALIVE socket option.

* TcpExtTCPSpuriousRTOs

The spurious retransmission timeout detected by the `F-RTO`_
algorithm.

.. _F-RTO: https://tools.ietf.org/html/rfc5682

TCP fast path와 slow path

TCP 수신에는 fast path와 slow path가 있습니다. zero window를 알렸거나 probing 중일 때, out-of-order segment가 왔을 때, urgent data를 기다릴 때, buffer 공간이 없을 때, 예상 밖 flag·window·header length나 TCP option이 왔을 때 fast path를 끕니다. 양방향 data처럼 sequence와 ACK 값이 모두 변하는 경우도 fast path가 지원하지 않습니다.

kernel은 조건이 없으면 fast path를 시도합니다. out-of-order packet과 delayed ACK는 slow path로 가므로 성능에 영향을 줄 수 있습니다. window scale을 쓰지 않으면 Established 진입 직후 fast path를 켜려 하지만, window scale을 쓰면 처음에는 끄고 packet을 받은 뒤 활성화를 시도합니다.

data 없는 ACK가 fast path에서 처리되면 `TcpExtTCPHPAcks`, slow path면 `TcpExtTCPPureAcks`가 증가합니다. data가 있는 packet을 fast path에서 처리하면 `TcpExtTCPHPHits`가 증가합니다.

TCP Fast Path
=============
When kernel receives a TCP packet, it has two paths to handler the
packet, one is fast path, another is slow path. The comment in kernel
code provides a good explanation of them, I pasted them below::

  It is split into a fast path and a slow path. The fast path is
  disabled when:

  - A zero window was announced from us
  - zero window probing
    is only handled properly on the slow path.
  - Out of order segments arrived.
  - Urgent data is expected.
  - There is no buffer space left
  - Unexpected TCP flags/window values/header lengths are received
    (detected by checking the TCP header against pred_flags)
  - Data is sent in both directions. The fast path only supports pure senders
    or pure receivers (this means either the sequence number or the ack
    value must stay constant)
  - Unexpected TCP option.

Kernel will try to use fast path unless any of the above conditions
are satisfied. If the packets are out of order, kernel will handle
them in slow path, which means the performance might be not very
good. Kernel would also come into slow path if the "Delayed ack" is
used, because when using "Delayed ack", the data is sent in both
directions. When the TCP window scale option is not used, kernel will
try to enable fast path immediately when the connection comes into the
established state, but if the TCP window scale option is used, kernel
will disable the fast path at first, and try to enable it after kernel
receives packets.

* TcpExtTCPPureAcks and TcpExtTCPHPAcks

If a packet set ACK flag and has no data, it is a pure ACK packet, if
kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
if kernel handles it in the slow path, TcpExtTCPPureAcks will
increase 1.

* TcpExtTCPHPHits

If a TCP packet has data (which means it is not a pure ACK packet),
and this packet is handled in the fast path, TcpExtTCPHPHits will
increase 1.

TCP abort counter

`TcpExtTCPAbortOnData`는 in-flight data가 있는데 연결을 닫아 RST를 보내는 비정상 종료입니다. 기본 `close()`는 즉시 반환하고 kernel이 남은 data를 비동기로 보냅니다. `SO_LINGER`에서 `l_onoff=1`, 양수 `l_linger`면 최대 해당 초 동안 ACK를 기다립니다. `l_linger=0`이면 즉시 RST를 보내고 이 counter를 증가시킵니다.

`TcpExtTCPAbortOnClose`는 응용이 읽지 않은 수신 data를 남긴 채 연결을 닫아 kernel이 상대에게 RST를 보낸 경우입니다. `TcpExtTCPAbortOnMemory`는 응용과 분리된 orphan socket을 유지할 메모리가 부족해 RST를 보내고 socket을 삭제한 횟수입니다. TCP memory가 `tcp_mem` 세 번째 값보다 높거나 orphan 수가 `net.ipv4.tcp_max_orphans`보다 높을 때 발생합니다.

`TcpExtTCPAbortOnTimeout`은 TCP timer가 만료되어 RST 없이 연결을 포기한 경우입니다. `TcpExtTCPAbortOnLinger`는 FIN_WAIT_2에서 상대 FIN을 기다리지 않고 `TCP_LINGER2` 정책에 따라 RST로 즉시 삭제한 횟수입니다. RFC2525 2.17 조건에서 RST를 보내는 내부 처리 자체가 실패하면 `TcpExtTCPAbortFailed`가 증가합니다.

TCP abort
=========
* TcpExtTCPAbortOnData

It means TCP layer has data in flight, but need to close the
connection. So TCP layer sends a RST to the other side, indicate the
connection is not closed very graceful. An easy way to increase this
counter is using the SO_LINGER option. Please refer to the SO_LINGER
section of the `socket man page`_:

.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html

By default, when an application closes a connection, the close function
will return immediately and kernel will try to send the in-flight data
async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
to a positive number, the close function won't return immediately, but
wait for the in-flight data are acked by the other side, the max wait
time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
when the application closes a connection, kernel will send a RST
immediately and increase the TcpExtTCPAbortOnData counter.

* TcpExtTCPAbortOnClose

This counter means the application has unread data in the TCP layer when
the application wants to close the TCP connection. In such a situation,
kernel will send a RST to the other side of the TCP connection.

* TcpExtTCPAbortOnMemory

When an application closes a TCP connection, kernel still need to track
the connection, let it complete the TCP disconnect process. E.g. an
app calls the close method of a socket, kernel sends fin to the other
side of the connection, then the app has no relationship with the
socket any more, but kernel need to keep the socket, this socket
becomes an orphan socket, kernel waits for the reply of the other side,
and would come to the TIME_WAIT state finally. When kernel has no
enough memory to keep the orphan socket, kernel would send an RST to
the other side, and delete the socket, in such situation, kernel will
increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
TcpExtTCPAbortOnMemory:

1. the memory used by the TCP protocol is higher than the third value of
the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:

.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html

2. the orphan socket count is higher than net.ipv4.tcp_max_orphans


* TcpExtTCPAbortOnTimeout

This counter will increase when any of the TCP timers expire. In such
situation, kernel won't send RST, just give up the connection.

* TcpExtTCPAbortOnLinger

When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
for the fin packet from the other side, kernel could send a RST and
delete the socket immediately. This is not the default behavior of
Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
you could let kernel follow this behavior.

* TcpExtTCPAbortFailed

The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
satisfied. If an internal error occurs during this process,
TcpExtTCPAbortFailed will be increased.

.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50

TCP Hybrid Slow Start

Hybrid Slow Start는 전통적 congestion window Slow Start를 개선하여 path 최대 bandwidth 접근 여부를 ACK train 길이와 packet delay 증가로 탐지합니다. 둘 중 하나가 임계값에 도달하면 Congestion Avoidance로 전환합니다. 문서 기준 v4.20까지 cubic과 cdg가 이를 사용합니다.

`TcpExtTCPHystartTrainDetect`는 ACK train 임계값 탐지 횟수이고 `TcpExtTCPHystartTrainCwnd`는 그때 CWND의 합입니다. 합을 탐지 횟수로 나누면 평균 CWND입니다. `TcpExtTCPHystartDelayDetect`와 `TcpExtTCPHystartDelayCwnd`는 packet delay 기준으로 같은 정보를 제공합니다.

TCP Hybrid Slow Start
=====================
The Hybrid Slow Start algorithm is an enhancement of the traditional
TCP congestion window Slow Start algorithm. It uses two pieces of
information to detect whether the max bandwidth of the TCP path is
approached. The two pieces of information are ACK train length and
increase in packet delay. For detail information, please refer the
`Hybrid Slow Start paper`_. Either ACK train length or packet delay
hits a specific threshold, the congestion control algorithm will come
into the Congestion Avoidance state. Until v4.20, two congestion
control algorithms are using Hybrid Slow Start, they are cubic (the
default congestion control algorithm) and cdg. Four snmp counters
relate with the Hybrid Slow Start algorithm.

.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf

* TcpExtTCPHystartTrainDetect

How many times the ACK train length threshold is detected

* TcpExtTCPHystartTrainCwnd

The sum of CWND detected by ACK train length. Dividing this value by
TcpExtTCPHystartTrainDetect is the average CWND which detected by the
ACK train length.

* TcpExtTCPHystartDelayDetect

How many times the packet delay threshold is detected.

* TcpExtTCPHystartDelayCwnd

The sum of CWND detected by packet delay. Dividing this value by
TcpExtTCPHystartDelayDetect is the average CWND which detected by the
packet delay.

재전송과 congestion control

TCP 재전송은 SACK과 fast recovery(Reno) 가운데 하나를 사용합니다. SACK option이 켜지면 SACK, 아니면 RFC6582 fast recovery를 사용합니다. congestion control state는 Open, Disorder, CWR, Recovery, Loss 다섯 가지입니다.

Recovery 진입 시 SACK을 쓰면 `TcpExtTCPSackRecovery`, 아니면 `TcpExtTCPRenoRecovery`가 증가하며 lost packet 재전송 시작을 뜻합니다. SACK로 ACK됐던 packet을 receiver가 버려 RTO 뒤 다시 전송해야 하면 `TcpExtTCPSACKReneging`이 증가합니다.

SACK이 꺼진 Reno에서 예상보다 duplicate ACK가 많아 재전송 원본이 loss가 아니라 reorder였음을 알면 `TcpExtTCPRenoReorder`입니다. hole을 채운 packet이 아직 재전송되지 않았거나 그 ACK timestamp가 재전송 전이면 `TcpExtTCPTSReorder`입니다. DSACK를 받거나 SACK block 순서로 미재전송 packet의 reorder를 알면 `TcpExtTCPSACKReorder`가 증가합니다.

Loss state에서 재전송하면 `TcpExtTCPSlowStartRetrans`, Loss가 아닌 상태면 `TcpExtTCPFastRetrans`입니다. SACK가 retransmitted packet의 재손실을 알리면 `TcpExtTCPLostRetransmit`, 하위 계층에 재전송 packet 전달을 시도했지만 오류가 나면 `TcpExtTCPRetransFail`, SYN을 재전송하면 `TcpExtTCPSynRetrans`입니다.

TCP retransmission and congestion control
=========================================
The TCP protocol has two retransmission mechanisms: SACK and fast
recovery. They are exclusive with each other. When SACK is enabled,
the kernel TCP stack would use SACK, or kernel would use fast
recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
the fast recovery is defined in `RFC6582`_, which is also called
'Reno'.

The TCP congestion control is a big and complex topic. To understand
the related snmp counter, we need to know the states of the congestion
control state machine. There are 5 states: Open, Disorder, CWR,
Recovery and Loss. For details about these states, please refer page 5
and page 6 of this document:
https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf

.. _RFC2018: https://tools.ietf.org/html/rfc2018
.. _RFC6582: https://tools.ietf.org/html/rfc6582

* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery

When the congestion control comes into Recovery state, if sack is
used, TcpExtTCPSackRecovery increases 1, if sack is not used,
TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
stack begins to retransmit the lost packets.

* TcpExtTCPSACKReneging

A packet was acknowledged by SACK, but the receiver has dropped this
packet, so the sender needs to retransmit this packet. In this
situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
could drop a packet which has been acknowledged by SACK, although it is
unusual, it is allowed by the TCP protocol. The sender doesn't really
know what happened on the receiver side. The sender just waits until
the RTO expires for this packet, then the sender assumes this packet
has been dropped by the receiver.

* TcpExtTCPRenoReorder

The reorder packet is detected by fast recovery. It would only be used
if SACK is disabled. The fast recovery algorithm detects recorder by
the duplicate ACK number. E.g., if retransmission is triggered, and
the original retransmitted packet is not lost, it is just out of
order, the receiver would acknowledge multiple times, one for the
retransmitted packet, another for the arriving of the original out of
order packet. Thus the sender would find more ACks than its
expectation, and the sender knows out of order occurs.

* TcpExtTCPTSReorder

The reorder packet is detected when a hole is filled. E.g., assume the
sender sends packet 1,2,3,4,5, and the receiving order is
1,2,4,5,3. When the sender receives the ACK of packet 3 (which will
fill the hole), two conditions will let TcpExtTCPTSReorder increase
1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
3 is retransmitted but the timestamp of the packet 3's ACK is earlier
than the retransmission timestamp.

* TcpExtTCPSACKReorder

The reorder packet detected by SACK. The SACK has two methods to
detect reorder: (1) DSACK is received by the sender. It means the
sender sends the same packet more than one times. And the only reason
is the sender believes an out of order packet is lost so it sends the
packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
the sender has received SACKs for packet 2 and 5, now the sender
receives SACK for packet 4 and the sender doesn't retransmit the
packet yet, the sender would know packet 4 is out of order. The TCP
stack of kernel will increase TcpExtTCPSACKReorder for both of the
above scenarios.

* TcpExtTCPSlowStartRetrans

The TCP stack wants to retransmit a packet and the congestion control
state is 'Loss'.

* TcpExtTCPFastRetrans

The TCP stack wants to retransmit a packet and the congestion control
state is not 'Loss'.

* TcpExtTCPLostRetransmit

A SACK points out that a retransmission packet is lost again.

* TcpExtTCPRetransFail

The TCP stack tries to deliver a retransmission packet to lower layers
but the lower layers return an error.

* TcpExtTCPSynRetrans

The TCP stack retransmits a SYN packet.

DSACK와 잘못된 SACK block

RFC2883 DSACK는 receiver가 duplicate packet을 sender에게 보고합니다. 이미 ACK된 packet의 duplicate를 받고 DSACK를 보내면 `TcpExtTCPDSACKOldSent`, out-of-order duplicate에 DSACK를 보내면 `TcpExtTCPDSACKOfoSent`입니다. sender가 각각의 DSACK를 받으면 `TcpExtTCPDSACKRecv`, `TcpExtTCPDSACKOfoRecv`가 증가합니다.

SACK/DSACK block 유효성은 start·end sequence number를 `tcp_is_sackblock_valid` 규칙으로 검사합니다. option 하나에 최대 네 block이 있고 각각 세므로 세 block이 잘못되면 counter도 세 번 증가합니다.

`TcpExtTCPSACKDiscard`는 invalid SACK block 수이지만 ACK recording 때문에 invalid가 된 block은 무시만 하고 세지 않습니다. invalid DSACK는 socket의 `undo_marker`가 없으면 중간 네트워크 duplicate 가능성을 뜻하는 `TcpExtTCPDSACKIgnoredNoUndo`, marker가 있으면 오래된 packet 가능성을 뜻하는 `TcpExtTCPDSACKIgnoredOld`로 기록합니다.

DSACK
=====
The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
duplicate packets to the sender. There are two kinds of
duplications: (1) a packet which has been acknowledged is
duplicate. (2) an out of order packet is duplicate. The TCP stack
counts these two kinds of duplications on both receiver side and
sender side.

.. _RFC2883 : https://tools.ietf.org/html/rfc2883

* TcpExtTCPDSACKOldSent

The TCP stack receives a duplicate packet which has been acked, so it
sends a DSACK to the sender.

* TcpExtTCPDSACKOfoSent

The TCP stack receives an out of order duplicate packet, so it sends a
DSACK to the sender.

* TcpExtTCPDSACKRecv

The TCP stack receives a DSACK, which indicates an acknowledged
duplicate packet is received.

* TcpExtTCPDSACKOfoRecv

The TCP stack receives a DSACK, which indicate an out of order
duplicate packet is received.

invalid SACK and DSACK
======================
When a SACK (or DSACK) block is invalid, a corresponding counter would
be updated. The validation method is base on the start/end sequence
number of the SACK block. For more details, please refer the comment
of the function tcp_is_sackblock_valid in the kernel source code. A
SACK option could have up to 4 blocks, they are checked
individually. E.g., if 3 blocks of a SACk is invalid, the
corresponding counter would be updated 3 times. The comment of commit
18f02545a9a1 ("[TCP] MIB: Add counters for discarded SACK blocks")
has additional explanation:

* TcpExtTCPSACKDiscard

This counter indicates how many SACK blocks are invalid. If the invalid
SACK block is caused by ACK recording, the TCP stack will only ignore
it and won't update this counter.

* TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo

When a DSACK block is invalid, one of these two counters would be
updated. Which counter will be updated depends on the undo_marker flag
of the TCP socket. If the undo_marker is not set, the TCP stack isn't
likely to re-transmit any packets, and we still receive an invalid
DSACK block, the reason might be that the packet is duplicated in the
middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
will be updated. As implied in its name, it might be an old packet.

SACK shift와 merge

SACK block이 여러 skb에 걸치면 TCP는 skb data를 재배치합니다. 예를 들어 ACK 범위 10~15가 skb1의 10~13과 skb2의 14~20에 걸치면 skb2의 14~15를 skb1로 옮기는 shift를 합니다. ACK 범위가 10~20 전체라면 skb2 data를 모두 skb1로 옮기고 skb2를 버리는 merge입니다.

skb가 shift되면 `TcpExtTCPSackShifted`, merge되면 `TcpExtTCPSackMerged`가 증가합니다. 필요했지만 어떤 이유로 실행하지 못하면 `TcpExtTCPSackShiftFallback`입니다.

SACK shift
==========
The linux networking stack stores data in sk_buff struct (skb for
short). If a SACK block acrosses multiple skb, the TCP stack will try
to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
15 in skb2 would be moved to skb1. This operation is 'shift'. If a
SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
discard, this operation is 'merge'.

* TcpExtTCPSackShifted

A skb is shifted

* TcpExtTCPSackMerged

A skb is merged

* TcpExtTCPSackShiftFallback

A skb should be shifted or merged, but the TCP stack doesn't do it for
some reasons.

Out-of-order queue와 PAWS

out-of-order packet을 받아 메모리가 충분해 queue에 넣으면 `TcpExtTCPOFOQueue`, 부족해 버리면 `TcpExtTCPOFODrop`입니다. 새 out-of-order packet이 이전 packet과 겹치면 중복 부분을 버리고 `TcpExtTCPOFOMerge`를 증가시키며, 이 packet은 `TCPOFOQueue`에도 함께 집계됩니다.

PAWS(Protection Against Wrapped Sequence numbers)는 TCP timestamp로 오래된 packet을 버립니다. SYN-SENT 상태에서 PAWS drop이면 `TcpExtPAWSActive`, 다른 상태면 `TcpExtPAWSEstab`입니다.

TCP out of order
================
* TcpExtTCPOFOQueue

The TCP layer receives an out of order packet and has enough memory
to queue it.

* TcpExtTCPOFODrop

The TCP layer receives an out of order packet but doesn't have enough
memory, so drops it. Such packets won't be counted into
TcpExtTCPOFOQueue.

* TcpExtTCPOFOMerge

The received out of order packet has an overlay with the previous
packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
packets will also be counted into TcpExtTCPOFOQueue.

TCP PAWS
========
PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
which is used to drop old packets. It depends on the TCP
timestamps. For detail information, please refer the `timestamp wiki`_
and the `RFC of PAWS`_.

.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps

* TcpExtPAWSActive

Packets are dropped by PAWS in Syn-Sent status.

* TcpExtPAWSEstab

Packets are dropped by PAWS in any status other than Syn-Sent.

tcp_invalid_ratelimit과 ACK 생략

kernel은 중복 ACK를 지나치게 자주 보내지 않도록 `tcp_invalid_ratelimit`에 따라 일부 ACK를 생략합니다. 받은 packet이 SYN이거나 data가 없을 때만 생략하며, 상황별 counter로 원인을 기록합니다.

`TcpExtTCPACKSkippedSynRecv`는 SYN-RECV에서 반복 SYN, PAWS 실패 또는 window 밖 sequence에 보낼 ACK가 rate limit을 넘은 경우입니다. PAWS 실패가 SYN-RECV, FIN-WAIT-2, TIME-WAIT면 각 상태 counter에 들어가고 그 밖의 상태면 `TcpExtTCPACKSkippedPAWS`입니다.

timestamp는 PAWS를 통과했지만 sequence가 window 밖이고 상태가 SYN-RECV·FIN-WAIT-2·TIME-WAIT가 아니면 `TcpExtTCPACKSkippedSeq`입니다. FIN-WAIT-2와 TIME-WAIT에서 PAWS 실패 또는 window 밖 sequence면 각각 `TcpExtTCPACKSkippedFinWait2`, `TcpExtTCPACKSkippedTimeWait`입니다.

RFC5961의 세 challenge ACK 상황과 Linux의 더 엄격한 ACK number 검사에서 challenge ACK를 생략하면 `TcpExtTCPACKSkippedChallenge`가 증가합니다.

TCP ACK skip
============
In some scenarios, kernel would avoid sending duplicate ACKs too
frequently. Please find more details in the tcp_invalid_ratelimit
section of the `sysctl document`_. When kernel decides to skip an ACK
due to tcp_invalid_ratelimit, kernel would update one of below
counters to indicate the ACK is skipped in which scenario. The ACK
would only be skipped if the received packet is either a SYN packet or
it has no data.

.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst

* TcpExtTCPACKSkippedSynRecv

The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
in the Syn-Recv status. But in several scenarios, the TCP stack need
to send an ACK. E.g., the TCP stack receives the same SYN packet
repeately, the received packet does not pass the PAWS check, or the
received packet sequence number is out of window. In these scenarios,
the TCP stack needs to send ACK. If the ACk sending frequency is higher than
tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
increase TcpExtTCPACKSkippedSynRecv.


* TcpExtTCPACKSkippedPAWS

The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
or Time-Wait statuses, the skipped ACK would be counted to
TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
would be counted to TcpExtTCPACKSkippedPAWS.

* TcpExtTCPACKSkippedSeq

The sequence number is out of window and the timestamp passes the PAWS
check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.

* TcpExtTCPACKSkippedFinWait2

The ACK is skipped in Fin-Wait-2 status, the reason would be either
PAWS check fails or the received sequence number is out of window.

* TcpExtTCPACKSkippedTimeWait

The ACK is skipped in Time-Wait status, the reason would be either
PAWS check failed or the received sequence number is out of window.

* TcpExtTCPACKSkippedChallenge

The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
three scenarios, In some TCP status, the linux TCP stack would also
send challenge ACKs if the ACK number is before the first
unacknowledged number (more strict than `RFC 5961 section 5.2`_).

.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11

수신 window와 Delayed ACK

메모리 상태상 receive window를 0으로 만들고 싶지만 이전 광고 window에서 이미 소비된 byte를 반영해 아직 0이 아닌 경우 `TcpExtTCPWantZeroWindowAdv`입니다. 예를 들어 이전 window 10에서 3byte를 받으면 계산 목표가 0이어도 현재 광고값은 7일 수 있습니다. 0이 아닌 값에서 실제 0으로 전환하면 `TcpExtTCPToZeroWindowAdv`, 0에서 다시 0이 아닌 값으로 열리면 `TcpExtTCPFromZeroWindowAdv`입니다.

Delayed ACK timer가 만료되어 pure ACK를 보내고 delayed mode를 끝내면 `TcpExtDelayedACKs`입니다. userspace가 socket lock을 잡아 즉시 보낼 수 없으면 `TcpExtDelayedACKLocked`가 먼저 증가하고, 나중에 unlock 뒤 보낼 때 `DelayedACKs`도 증가합니다. 이미 ACK한 packet을 다시 받으면 `TcpExtDelayedACKLost`인데 delayed ACK 손실 외에 네트워크 duplicate도 원인이 될 수 있습니다.

TCP receive window
==================
* TcpExtTCPWantZeroWindowAdv

Depending on current memory usage, the TCP stack tries to set receive
window to zero. But the receive window might still be a no-zero
value. For example, if the previous window size is 10, and the TCP
stack receives 3 bytes, the current window size would be 7 even if the
window size calculated by the memory usage is zero.

* TcpExtTCPToZeroWindowAdv

The TCP receive window is set to zero from a no-zero value.

* TcpExtTCPFromZeroWindowAdv

The TCP receive window is set to no-zero value from zero.


Delayed ACK
===========
The TCP Delayed ACK is a technique which is used for reducing the
packet count in the network. For more details, please refer the
`Delayed ACK wiki`_

.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment

* TcpExtDelayedACKs

A delayed ACK timer expires. The TCP stack will send a pure ACK packet
and exit the delayed ACK mode.

* TcpExtDelayedACKLocked

A delayed ACK timer expires, but the TCP stack can't send an ACK
immediately due to the socket is locked by a userspace program. The
TCP stack will send a pure ACK later (after the userspace program
unlock the socket). When the TCP stack sends the pure ACK later, the
TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
mode.

* TcpExtDelayedACKLost

It will be updated when the TCP stack receives a packet which has been
ACKed. A Delayed ACK loss might cause this issue, but it would also be
triggered by other reasons, such as a packet is duplicated in the
network.

Tail Loss Probe와 TCP Fast Open

TLP는 TCP tail packet loss를 탐지하는 알고리즘입니다. probe packet을 보내면 `TcpExtTCPLossProbes`, TLP가 loss를 탐지하고 복구하면 `TcpExtTCPLossProbeRecovery`가 증가합니다.

TCP Fast Open은 3-way handshake 완료 전 data 전송을 허용합니다. SYN-SENT에서 ACK가 SYN data까지 확인해 TFO cookie 수락을 알리면 `TcpExtTCPFastOpenActive`입니다. 상대가 SYN data를 ACK하지 않거나, cookie가 든 SYN이 한 번 이상 timeout되거나, handshake 후 middlebox black-hole 때문에 RTO가 `tcp_retries1`번 발생하면 `TcpExtTCPFastOpenActiveFail`입니다.

서버가 Fast Open을 수락하면 `TcpExtTCPFastOpenPassive`, cookie가 invalid하거나 socket 생성 오류로 거절하면 `TcpExtTCPFastOpenPassiveFail`입니다. pending request가 `fastopenq->max_qlen`보다 많으면 `TcpExtTCPFastOpenListenOverflow`만 증가하고 passive·passive fail에는 들어가지 않습니다. 이 길이는 `TCP_FASTOPEN` setsockopt로 정하며 `net.core.somaxconn`을 넘을 수 없습니다. client의 TFO cookie 요청은 `TcpExtTCPFastOpenCookieReqd`입니다.

Tail Loss Probe (TLP)
=====================
TLP is an algorithm which is used to detect TCP packet loss. For more
details, please refer the `TLP paper`_.

.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01

* TcpExtTCPLossProbes

A TLP probe packet is sent.

* TcpExtTCPLossProbeRecovery

A packet loss is detected and recovered by TLP.

TCP Fast Open description
=========================
TCP Fast Open is a technology which allows data transfer before the
3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
general description.

.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open

* TcpExtTCPFastOpenActive

When the TCP stack receives an ACK packet in the SYN-SENT status, and
the ACK packet acknowledges the data in the SYN packet, the TCP stack
understand the TFO cookie is accepted by the other side, then it
updates this counter.

* TcpExtTCPFastOpenActiveFail

This counter indicates that the TCP stack initiated a TCP Fast Open,
but it failed. This counter would be updated in three scenarios: (1)
the other side doesn't acknowledge the data in the SYN packet. (2) The
SYN packet which has the TFO cookie is timeout at least once. (3)
after the 3-way handshake, the retransmission timeout happens
net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
fast open after the handshake.

* TcpExtTCPFastOpenPassive

This counter indicates how many times the TCP stack accepts the fast
open request.

* TcpExtTCPFastOpenPassiveFail

This counter indicates how many times the TCP stack rejects the fast
open request. It is caused by either the TFO cookie is invalid or the
TCP stack finds an error during the socket creating process.

* TcpExtTCPFastOpenListenOverflow

When the pending fast open request number is larger than
fastopenq->max_qlen, the TCP stack will reject the fast open request
and update this counter. When this counter is updated, the TCP stack
won't update TcpExtTCPFastOpenPassive or
TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
TCP_FASTOPEN socket operation and it could not be larger than
net.core.somaxconn. For example:

setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));

* TcpExtTCPFastOpenCookieReqd

This counter indicates how many times a client wants to request a TFO
cookie.

SYN cookie와 challenge ACK

SYN cookie는 SYN flood를 완화합니다. 송신한 cookie 수는 `TcpExtSyncookiesSent`, cookie 응답 수신은 `TcpExtSyncookiesRecv`입니다. cookie에서 decode한 MSS가 invalid면 `TcpExtSyncookiesFailed`가 증가하고 이 packet은 cookie 응답으로 취급하지 않아 `SyncookiesRecv`는 증가하지 않습니다.

보낸 challenge ACK 수는 `TcpExtTCPChallengeACK`입니다. SYN에 응답해 challenge ACK가 필요해진 횟수는 `TcpExtTCPSYNChallenge`이며, 그 뒤 실제 전송해 ChallengeACK를 증가시키거나 rate limit 때문에 생략해 `TcpExtTCPACKSkippedChallenge`를 증가시킬 수 있습니다.

SYN cookies
===========
SYN cookies are used to mitigate SYN flood, for details, please refer
the `SYN cookies wiki`_.

.. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies

* TcpExtSyncookiesSent

It indicates how many SYN cookies are sent.

* TcpExtSyncookiesRecv

How many reply packets of the SYN cookies the TCP stack receives.

* TcpExtSyncookiesFailed

The MSS decoded from the SYN cookie is invalid. When this counter is
updated, the received packet won't be treated as a SYN cookie and the
TcpExtSyncookiesRecv counter won't be updated.

Challenge ACK
=============
For details of challenge ACK, please refer the explanation of
TcpExtTCPACKSkippedChallenge.

* TcpExtTCPChallengeACK

The number of challenge acks sent.

* TcpExtTCPSYNChallenge

The number of challenge acks sent in response to SYN packets. After
updates this counter, the TCP stack might send a challenge ACK and
update the TcpExtTCPChallengeACK counter, or it might also skip to
send the challenge and update the TcpExtTCPACKSkippedChallenge.

메모리 압박과 receive queue prune

socket이 메모리 압박을 받으면 TCP는 receive queue와 out-of-order queue에서 메모리를 회수합니다. collapse는 큰 skb를 새로 할당해 연속 skb의 내용을 복사하고 기존 skb들을 해제하는 방식입니다.

회수를 시도하면 `TcpExtPruneCalled`가 증가하고 먼저 두 queue를 collapse합니다. 여전히 부족하면 out-of-order queue packet을 버리며 `TcpExtOfoPruned`가 증가합니다. 그래도 실제 사용량이 허용 최대보다 크면 prune 실패를 뜻하는 `TcpExtRcvPruned`가 증가합니다. collapse에서 해제한 skb 수는 `TcpExtTCPRcvCollapsed`입니다.

prune
=====
When a socket is under memory pressure, the TCP stack will try to
reclaim memory from the receiving queue and out of order queue. One of
the reclaiming method is 'collapse', which means allocate a big skb,
copy the contiguous skbs to the single big skb, and free these
contiguous skbs.

* TcpExtPruneCalled

The TCP stack tries to reclaim memory for a socket. After updates this
counter, the TCP stack will try to collapse the out of order queue and
the receiving queue. If the memory is still not enough, the TCP stack
will try to discard packets from the out of order queue (and update the
TcpExtOfoPruned counter)

* TcpExtOfoPruned

The TCP stack tries to discard packet on the out of order queue.

* TcpExtRcvPruned

After 'collapse' and discard packets from the out of order queue, if
the actually used memory is still larger than the max allowed memory,
this counter will be updated. It means the 'prune' fails.

* TcpExtTCPRcvCollapsed

This counter indicates how many skbs are freed during 'collapse'.

예제: ping counter

8.8.8.8에 ping 한 번을 보내면 송신 Echo 때문에 `IpOutRequests`, `IcmpOutMsgs`, `IcmpOutEchos`, `IcmpMsgOutType8`이 각각 1 증가합니다. Echo Reply 수신은 `IpInReceives`, `IcmpInMsgs`, `IcmpInEchoReps`, `IcmpMsgInType0`을 1 증가시키고 ICMP 계층으로 전달되므로 `IpInDelivers`도 1 증가합니다.

기본 ping data 48byte에 IP header 20byte와 ICMP header 16byte를 더하면 IP 계층이 세는 크기는 84byte입니다. MAC header 14byte는 `IpExtInOctets`와 `IpExtOutOctets`에서 제외되므로 두 값이 84가 됩니다.

examples
========

ping test
---------
Run the ping command against the public dns server 8.8.8.8::

  nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
  PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
  64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms

  --- 8.8.8.8 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms

The nstayt result::

  nstatuser@nstat-a:~$ nstat
  #kernel
  IpInReceives                    1                  0.0
  IpInDelivers                    1                  0.0
  IpOutRequests                   1                  0.0
  IcmpInMsgs                      1                  0.0
  IcmpInEchoReps                  1                  0.0
  IcmpOutMsgs                     1                  0.0
  IcmpOutEchos                    1                  0.0
  IcmpMsgInType0                  1                  0.0
  IcmpMsgOutType8                 1                  0.0
  IpExtInOctets                   84                 0.0
  IpExtOutOctets                  84                 0.0
  IpExtInNoECTPkts                1                  0.0

The Linux server sent an ICMP Echo packet, so IpOutRequests,
IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
was passed to the ICMP layer via IP layer, so IpInDelivers was
increased 1. The default ping data size is 48, so an ICMP Echo packet
and its corresponding Echo Reply packet are constructed by:

* 14 bytes MAC header
* 20 bytes IP header
* 16 bytes ICMP header
* 48 bytes data (default value of the ping command)

So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.

예제: TCP 3-way handshake

server가 TCP 9000에서 listen하고 client가 연결하면 server는 SYN을 받고 SYN+ACK를 보내 `SYN-RCVD`가 되므로 `TcpPassiveOpens=1`입니다. SYN과 마지막 ACK 두 packet을 받고 SYN+ACK 하나를 보내 `TcpInSegs=2`, `TcpOutSegs=1`이며 마지막 pure ACK가 slow path에서 처리되어 `TcpExtTCPPureAcks=1`입니다.

client는 SYN을 보내 `SYN-SENT`가 되므로 `TcpActiveOpens=1`입니다. SYN과 마지막 ACK 두 packet을 보내고 SYN+ACK 하나를 받아 `TcpOutSegs=2`, `TcpInSegs=1`입니다.

tcp 3-way handshake
-------------------
On server side, we run::

  nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
  Listening on [0.0.0.0] (family 0, port 9000)

On client side, we run::

  nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
  Connection to 192.168.122.251 9000 port [tcp/*] succeeded!

The server listened on tcp 9000 port, the client connected to it, they
completed the 3-way handshake.

On server side, we can find below nstat output::

  nstatuser@nstat-b:~$ nstat | grep -i tcp
  TcpPassiveOpens                 1                  0.0
  TcpInSegs                       2                  0.0
  TcpOutSegs                      1                  0.0
  TcpExtTCPPureAcks               1                  0.0

On client side, we can find below nstat output::

  nstatuser@nstat-a:~$ nstat | grep -i tcp
  TcpActiveOpens                  1                  0.0
  TcpInSegs                       1                  0.0
  TcpOutSegs                      2                  0.0

When the server received the first SYN, it replied a SYN+ACK, and came into
SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
of the 3-way handshake is a pure ACK without data, so
TcpExtTCPPureAcks increased 1.

When the client sent SYN, the client came into the SYN-SENT state, so
TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
1, TcpOutSegs increased 2.

예제: normal TCP traffic과 fast path

nc 연결에서 client가 `hello`를 보내면 client는 data packet 하나를 보내고 ACK 하나를 받습니다. `TcpExtTCPOrigDataSent=1`이고 첫 ACK는 아직 fast path가 활성화되지 않아 `TcpExtTCPPureAcks=1`입니다. server는 data 하나를 받고 ACK 하나를 보냅니다. IP octet은 ACK 52byte, 6byte payload가 있는 data packet은 58byte로 나타납니다.

같은 연결에서 `world`를 다시 보내면 트래픽 형태는 같지만 client의 ACK가 fast path에 들어가 `TcpExtTCPHPAcks=1`, server의 data packet은 `TcpExtTCPHPHits=1`로 집계됩니다. 첫 측정에 없던 fast-path counter가 두 번째에 나타나는 이유입니다.

`ss -o state established -i` 출력의 `wscale:7,7`은 양쪽 window scale이 7임을 뜻합니다. window scale을 사용하면 Established 진입 직후 fast path가 꺼져 있고 packet을 받은 뒤 활성화를 시도하므로 첫 교환은 slow path, 다음 교환은 fast path가 될 수 있습니다.

TCP normal traffic
------------------
Run nc on server::

  nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
  Listening on [0.0.0.0] (family 0, port 9000)

Run nc on client::

  nstatuser@nstat-a:~$ nc -v nstat-b 9000
  Connection to nstat-b 9000 port [tcp/*] succeeded!

Input a string in the nc client ('hello' in our example)::

  nstatuser@nstat-a:~$ nc -v nstat-b 9000
  Connection to nstat-b 9000 port [tcp/*] succeeded!
  hello

The client side nstat output::

  nstatuser@nstat-a:~$ nstat
  #kernel
  IpInReceives                    1                  0.0
  IpInDelivers                    1                  0.0
  IpOutRequests                   1                  0.0
  TcpInSegs                       1                  0.0
  TcpOutSegs                      1                  0.0
  TcpExtTCPPureAcks               1                  0.0
  TcpExtTCPOrigDataSent           1                  0.0
  IpExtInOctets                   52                 0.0
  IpExtOutOctets                  58                 0.0
  IpExtInNoECTPkts                1                  0.0

The server side nstat output::

  nstatuser@nstat-b:~$ nstat
  #kernel
  IpInReceives                    1                  0.0
  IpInDelivers                    1                  0.0
  IpOutRequests                   1                  0.0
  TcpInSegs                       1                  0.0
  TcpOutSegs                      1                  0.0
  IpExtInOctets                   58                 0.0
  IpExtOutOctets                  52                 0.0
  IpExtInNoECTPkts                1                  0.0

Input a string in nc client side again ('world' in our example)::

  nstatuser@nstat-a:~$ nc -v nstat-b 9000
  Connection to nstat-b 9000 port [tcp/*] succeeded!
  hello
  world

Client side nstat output::

  nstatuser@nstat-a:~$ nstat
  #kernel
  IpInReceives                    1                  0.0
  IpInDelivers                    1                  0.0
  IpOutRequests                   1                  0.0
  TcpInSegs                       1                  0.0
  TcpOutSegs                      1                  0.0
  TcpExtTCPHPAcks                 1                  0.0
  TcpExtTCPOrigDataSent           1                  0.0
  IpExtInOctets                   52                 0.0
  IpExtOutOctets                  58                 0.0
  IpExtInNoECTPkts                1                  0.0


Server side nstat output::

  nstatuser@nstat-b:~$ nstat
  #kernel
  IpInReceives                    1                  0.0
  IpInDelivers                    1                  0.0
  IpOutRequests                   1                  0.0
  TcpInSegs                       1                  0.0
  TcpOutSegs                      1                  0.0
  TcpExtTCPHPHits                 1                  0.0
  IpExtInOctets                   58                 0.0
  IpExtOutOctets                  52                 0.0
  IpExtInNoECTPkts                1                  0.0

Compare the first client-side nstat and the second client-side nstat,
we could find one difference: the first one had a 'TcpExtTCPPureAcks',
but the second one had a 'TcpExtTCPHPAcks'. The first server-side
nstat and the second server-side nstat had a difference too: the
second server-side nstat had a TcpExtTCPHPHits, but the first
server-side nstat didn't have it. The network traffic patterns were
exactly the same: the client sent a packet to the server, the server
replied an ACK. But kernel handled them in different ways. When the
TCP window scale option is not used, kernel will try to enable fast
path immediately when the connection comes into the established state,
but if the TCP window scale option is used, kernel will disable the
fast path at first, and try to enable it after kernel receives
packets. We could use the 'ss' command to verify whether the window
scale option is used. e.g. run below command on either server or
client::

  nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
  Netid    Recv-Q     Send-Q            Local Address:Port             Peer Address:Port
  tcp      0          0               192.168.122.250:40654         192.168.122.251:9000
             ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98

The 'wscale:7,7' means both server and client set the window scale
option to 7. Now we could explain the nstat output in our test:

In the first nstat output of client side, the client sent a packet, server
reply an ACK, when kernel handled this ACK, the fast path was not
enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.

In the second nstat output of client side, the client sent a packet again,
and received another ACK from the server, in this time, the fast path is
enabled, and the ACK was qualified for fast path, so it was handled by
the fast path, so this ACK was counted into TcpExtTCPHPAcks.

In the first nstat output of server side, fast path was not enabled,
so there was no 'TcpExtTCPHPHits'.

In the second nstat output of server side, the fast path was enabled,
and the packet received from client qualified for fast path, so it
was counted into 'TcpExtTCPHPHits'.

예제: TCPAbortOnClose

server 예제는 TCP 9000 연결을 accept한 뒤 아무 data도 읽지 않습니다. client가 `hello`를 보내면 TCP 계층은 ACK하지만 응용 receive queue에는 읽지 않은 data가 남습니다. 이 상태에서 Ctrl-C로 server를 종료하면 kernel이 RST를 보내고 server의 `TcpExtTCPAbortOnClose`가 1 증가합니다. tcpdump로 RST를 확인할 수 있습니다.

TcpExtTCPAbortOnClose
---------------------
On the server side, we run below python script::

  import socket
  import time

  port = 9000

  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  s.bind(('0.0.0.0', port))
  s.listen(1)
  sock, addr = s.accept()
  while True:
      time.sleep(9999999)

This python script listen on 9000 port, but doesn't read anything from
the connection.

On the client side, we send the string "hello" by nc::

  nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000

Then, we come back to the server side, the server has received the "hello"
packet, and the TCP layer has acked this packet, but the application didn't
read it yet. We type Ctrl-C to terminate the server script. Then we
could find TcpExtTCPAbortOnClose increased 1 on the server side::

  nstatuser@nstat-b:~$ nstat | grep -i abort
  TcpExtTCPAbortOnClose           1                  0.0

If we run tcpdump on the server side, we could find the server sent a
RST after we type Ctrl-C.

예제: AbortOnMemory와 AbortOnTimeout

client의 `tcp_max_orphans`를 10으로 낮추고 server와 64개 연결을 만듭니다. server INPUT에서 TCP 9000을 drop한 뒤 client 프로그램을 종료하면 client는 연결을 닫으려 하지만 FIN이 server에 도달하지 않아 FIN_WAIT_1의 orphan socket으로 남습니다.

kernel은 orphan을 10개만 유지하고 나머지 54개에는 RST를 보낸 뒤 삭제하므로 `TcpExtTCPAbortOnMemory=54`, `ss -s`의 orphaned 값은 10이 됩니다. 성능상 kernel은 먼저 approximate orphan count를 확인하고 임계값을 넘을 때만 exact count를 재확인하므로 아주 작은 `tcp_max_orphans` 시험에서는 11개 연결만으로 counter가 기대대로 증가하지 않을 수 있습니다.

server firewall이 계속 FIN을 막으면 남은 10개 orphan도 FIN_WAIT_1에서 timer 만료로 포기됩니다. 몇 분 뒤 `TcpExtTCPAbortOnTimeout=10`이 됩니다.

TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
---------------------------------------------------
Below is an example which let the orphan socket count be higher than
net.ipv4.tcp_max_orphans.
Change tcp_max_orphans to a smaller value on client::

  sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"

Client code (create 64 connection to server)::

  nstatuser@nstat-a:~$ cat client_orphan.py
  import socket
  import time

  server = 'nstat-b' # server address
  port = 9000

  count = 64

  connection_list = []

  for i in range(64):
      s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      s.connect((server, port))
      connection_list.append(s)
      print("connection_count: %d" % len(connection_list))

  while True:
      time.sleep(99999)

Server code (accept 64 connection from client)::

  nstatuser@nstat-b:~$ cat server_orphan.py
  import socket
  import time

  port = 9000
  count = 64

  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  s.bind(('0.0.0.0', port))
  s.listen(count)
  connection_list = []
  while True:
      sock, addr = s.accept()
      connection_list.append((sock, addr))
      print("connection_count: %d" % len(connection_list))

Run the python scripts on server and client.

On server::

  python3 server_orphan.py

On client::

  python3 client_orphan.py

Run iptables on server::

  sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP

Type Ctrl-C on client, stop client_orphan.py.

Check TcpExtTCPAbortOnMemory on client::

  nstatuser@nstat-a:~$ nstat | grep -i abort
  TcpExtTCPAbortOnMemory          54                 0.0

Check orphaned socket count on client::

  nstatuser@nstat-a:~$ ss -s
  Total: 131 (kernel 0)
  TCP:   14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0

  Transport Total     IP        IPv6
  *         0         -         -
  RAW       1         0         1
  UDP       1         1         0
  TCP       14        13        1
  INET      16        14        2
  FRAG      0         0         0

The explanation of the test: after run server_orphan.py and
client_orphan.py, we set up 64 connections between server and
client. Run the iptables command, the server will drop all packets from
the client, type Ctrl-C on client_orphan.py, the system of the client
would try to close these connections, and before they are closed
gracefully, these connections became orphan sockets. As the iptables
of the server blocked packets from the client, the server won't receive fin
from the client, so all connection on clients would be stuck on FIN_WAIT_1
stage, so they will keep as orphan sockets until timeout. We have echo
10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
only keep 10 orphan sockets, for all other orphan sockets, the client
system sent RST for them and delete them. We have 64 connections, so
the 'ss -s' command shows the system has 10 orphan sockets, and the
value of TcpExtTCPAbortOnMemory was 54.

An additional explanation about orphan socket count: You could find the
exactly orphan socket count by the 'ss -s' command, but when kernel
decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
doesn't always check the exactly orphan socket count. For increasing
performance, kernel checks an approximate count firstly, if the
approximate count is more than tcp_max_orphans, kernel checks the
exact count again. So if the approximate count is less than
tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
would find TcpExtTCPAbortOnMemory is not increased at all. If
tcp_max_orphans is large enough, it won't occur, but if you decrease
tcp_max_orphans to a small value like our test, you might find this
issue. So in our test, the client set up 64 connections although the
tcp_max_orphans is 10. If the client only set up 11 connections, we
can't find the change of TcpExtTCPAbortOnMemory.

Continue the previous test, we wait for several minutes. Because of the
iptables on the server blocked the traffic, the server wouldn't receive
fin, and all the client's orphan sockets would timeout on the
FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
10 timeout on the client::

  nstatuser@nstat-a:~$ nstat | grep -i abort
  TcpExtTCPAbortOnTimeout         10                 0.0

예제: TCPAbortOnLinger

client는 `SO_LINGER=(1,10)`과 `TCP_LINGER2=-1`을 설정하고 연결 직후 닫습니다. server는 accept 뒤 읽지 않고 유지합니다. FIN_WAIT_2에서 linger 정책이 즉시 RST와 socket 삭제를 선택하여 client의 `TcpExtTCPAbortOnLinger`가 1 증가합니다.

TcpExtTCPAbortOnLinger
----------------------
The server side code::

  nstatuser@nstat-b:~$ cat server_linger.py
  import socket
  import time

  port = 9000

  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  s.bind(('0.0.0.0', port))
  s.listen(1)
  sock, addr = s.accept()
  while True:
      time.sleep(9999999)

The client side code::

  nstatuser@nstat-a:~$ cat client_linger.py
  import socket
  import struct

  server = 'nstat-b' # server address
  port = 9000

  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
  s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
  s.connect((server, port))
  s.close()

Run server_linger.py on server::

  nstatuser@nstat-b:~$ python3 server_linger.py

Run client_linger.py on client::

  nstatuser@nstat-a:~$ python3 client_linger.py

After run client_linger.py, check the output of nstat::

  nstatuser@nstat-a:~$ nstat | grep -i abort
  TcpExtTCPAbortOnLinger          1                  0.0

예제: TCPRcvCoalesce

server는 TCP 9000에서 연결을 받지만 data를 읽지 않습니다. client가 interactive socket에서 `foo`와 `bar`를 별도 3byte send로 보냅니다. 두 번째 packet이 올 때 첫 packet이 receive queue에 남아 있어 TCP가 둘을 합치며 server의 `TcpExtTCPRcvCoalesce=1`이 됩니다.

TcpExtTCPRcvCoalesce
--------------------
On the server, we run a program which listen on TCP port 9000, but
doesn't read any data::

  import socket
  import time
  port = 9000
  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  s.bind(('0.0.0.0', port))
  s.listen(1)
  sock, addr = s.accept()
  while True:
      time.sleep(9999999)

Save the above code as server_coalesce.py, and run::

  python3 server_coalesce.py

On the client, save below code as client_coalesce.py::

  import socket
  server = 'nstat-b'
  port = 9000
  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  s.connect((server, port))

Run::

  nstatuser@nstat-a:~$ python3 -i client_coalesce.py

We use '-i' to come into the interactive mode, then a packet::

  >>> s.send(b'foo')
  3

Send a packet again::

  >>> s.send(b'bar')
  3

On the server, run nstat::

  ubuntu@nstat-b:~$ nstat
  #kernel
  IpInReceives                    2                  0.0
  IpInDelivers                    2                  0.0
  IpOutRequests                   2                  0.0
  TcpInSegs                       2                  0.0
  TcpOutSegs                      2                  0.0
  TcpExtTCPRcvCoalesce            1                  0.0
  IpExtInOctets                   110                0.0
  IpExtOutOctets                  104                0.0
  IpExtInNoECTPkts                2                  0.0

The client sent two packets, server didn't read any data. When
the second packet arrived at server, the first packet was still in
the receiving queue. So the TCP layer merged the two packets, and we
could find the TcpExtTCPRcvCoalesce increased 1.

예제: ListenOverflows와 ListenDrops

nc server의 accept queue 설정 길이는 1이고 현재 Linux 구현의 실제 queue 길이는 설정값+1입니다. 첫 연결은 응용이 accept하고 다음 두 연결이 queue를 채웁니다. 네 번째 client가 연결하면 kernel 4.10 이상은 가득 찬 accept queue 때문에 SYN을 drop합니다.

시험의 `TcpExtListenOverflows`와 `TcpExtListenDrops`가 각각 4인 이유는 네 번째 client가 SYN 응답을 받지 못해 재시도했기 때문입니다. 기다리는 시간이 길면 두 값도 더 커집니다. 구형 kernel은 handshake를 완료한 뒤 half-open queue에 보관하는 동작을 보일 수 있습니다.

TcpExtListenOverflows and TcpExtListenDrops
-------------------------------------------
On server, run the nc command, listen on port 9000::

  nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
  Listening on [0.0.0.0] (family 0, port 9000)

On client, run 3 nc commands in different terminals::

  nstatuser@nstat-a:~$ nc -v nstat-b 9000
  Connection to nstat-b 9000 port [tcp/*] succeeded!

The nc command only accepts 1 connection, and the accept queue length
is 1. On current linux implementation, set queue length to n means the
actual queue length is n+1. Now we create 3 connections, 1 is accepted
by nc, 2 in accepted queue, so the accept queue is full.

Before running the 4th nc, we clean the nstat history on the server::

  nstatuser@nstat-b:~$ nstat -n

Run the 4th nc on the client::

  nstatuser@nstat-a:~$ nc -v nstat-b 9000

If the nc server is running on kernel 4.10 or higher version, you
won't see the "Connection to ... succeeded!" string, because kernel
will drop the SYN if the accept queue is full. If the nc client is running
on an old kernel, you would see that the connection is succeeded,
because kernel would complete the 3 way handshake and keep the socket
on half open queue. I did the test on kernel 4.15. Below is the nstat
on the server::

  nstatuser@nstat-b:~$ nstat
  #kernel
  IpInReceives                    4                  0.0
  IpInDelivers                    4                  0.0
  TcpInSegs                       4                  0.0
  TcpExtListenOverflows           4                  0.0
  TcpExtListenDrops               4                  0.0
  IpExtInOctets                   240                0.0
  IpExtInNoECTPkts                4                  0.0

Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
between the 4th nc and the nstat was longer, the value of
TcpExtListenOverflows and TcpExtListenDrops would be larger, because
the SYN of the 4th nc was dropped, the client was retrying.

예제: 세 가지 IPv4 route 오류

server A가 8.8.8.8을 server B 경유로 보내도록 route를 추가하고 B의 ICMP redirect를 끕니다. 첫 시험에서 B의 IP forwarding을 끄면 목적지가 local이 아닌 packet을 받아 drop하므로 `IpInAddrErrors`가 증가합니다. nc의 SYN 재시도 때문에 여러 번 증가할 수 있습니다.

두 번째 시험에서 B의 forwarding을 켜고 default route를 삭제합니다. B는 8.8.8.8 packet을 forward하려 하지만 route를 찾지 못해 `IpExtInNoRoutes=1`을 증가시키고 A에 ICMP Destination Unreachable type 3을 보냅니다. 이에 따라 `IcmpOutMsgs`, `IcmpOutDestUnreachs`, `IcmpMsgOutType3`도 증가합니다.

세 번째 시험은 default route가 없는 B 자체에서 8.8.8.8로 ping합니다. 송신 경로가 route를 찾지 못하므로 `IpOutNoRoutes=1`입니다.

IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
-------------------------------------------------
server A IP address: 192.168.122.250
server B IP address: 192.168.122.251
Prepare on server A, add a route to server B::

  $ sudo ip route add 8.8.8.8/32 via 192.168.122.251

Prepare on server B, disable send_redirects for all interfaces::

  $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
  $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
  $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
  $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0

We want to let sever A send a packet to 8.8.8.8, and route the packet
to server B. When server B receives such packet, it might send a ICMP
Redirect message to server A, set send_redirects to 0 will disable
this behavior.

First, generate InAddrErrors. On server B, we disable IP forwarding::

  $ sudo sysctl -w net.ipv4.conf.all.forwarding=0

On server A, we send packets to 8.8.8.8::

  $ nc -v 8.8.8.8 53

On server B, we check the output of nstat::

  $ nstat
  #kernel
  IpInReceives                    3                  0.0
  IpInAddrErrors                  3                  0.0
  IpExtInOctets                   180                0.0
  IpExtInNoECTPkts                3                  0.0

As we have let server A route 8.8.8.8 to server B, and we disabled IP
forwarding on server B, Server A sent packets to server B, then server B
dropped packets and increased IpInAddrErrors. As the nc command would
re-send the SYN packet if it didn't receive a SYN+ACK, we could find
multiple IpInAddrErrors.

Second, generate IpExtInNoRoutes. On server B, we enable IP
forwarding::

  $ sudo sysctl -w net.ipv4.conf.all.forwarding=1

Check the route table of server B and remove the default route::

  $ ip route show
  default via 192.168.122.1 dev ens3 proto static
  192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
  $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static

On server A, we contact 8.8.8.8 again::

  $ nc -v 8.8.8.8 53
  nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable

On server B, run nstat::

  $ nstat
  #kernel
  IpInReceives                    1                  0.0
  IpOutRequests                   1                  0.0
  IcmpOutMsgs                     1                  0.0
  IcmpOutDestUnreachs             1                  0.0
  IcmpMsgOutType3                 1                  0.0
  IpExtInNoRoutes                 1                  0.0
  IpExtInOctets                   60                 0.0
  IpExtOutOctets                  88                 0.0
  IpExtInNoECTPkts                1                  0.0

We enabled IP forwarding on server B, when server B received a packet
which destination IP address is 8.8.8.8, server B will try to forward
this packet. We have deleted the default route, there was no route for
8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
Destination Unreachable" message to server A.

Third, generate IpOutNoRoutes. Run ping command on server B::

  $ ping -c 1 8.8.8.8
  connect: Network is unreachable

Run nstat on server B::

  $ nstat
  #kernel
  IpOutNoRoutes                   1                  0.0

We have deleted the default route on server B. Server B couldn't find
a route for the 8.8.8.8 IP address, so server B increased
IpOutNoRoutes.

예제: TCPACKSkippedSynRecv

같은 SYN을 server에 세 번 보냅니다. 첫 SYN은 SYN-RECV socket과 SYN/ACK를 만들고, 두 번째는 SYN/ACK를 다시 보내 duplicate ACK 응답 시각을 기록합니다. 세 번째는 직전 응답 시각과 rate limit을 검사해 duplicate ACK를 생략하고 `TcpExtTCPACKSkippedSynRecv`를 증가시킵니다.

tcpdump로 SYN을 저장하고 hardware checksum offload 때문에 pcap checksum이 틀릴 수 있어 `tcprewrite --fixcsum`으로 고칩니다. client가 server의 SYN/ACK에 RST를 보내지 않도록 INPUT에서 source port 9000을 drop하고 `tcpreplay`로 SYN을 세 번 보낸 뒤 server에서 counter 1을 확인합니다.

TcpExtTCPACKSkippedSynRecv
--------------------------
In this test, we send 3 same SYN packets from client to server. The
first SYN will let server create a socket, set it to Syn-Recv status,
and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
again, and record the reply time (the duplicate ACK reply time). The
third SYN will let server check the previous duplicate ACK reply time,
and decide to skip the duplicate ACK, then increase the
TcpExtTCPACKSkippedSynRecv counter.

Run tcpdump to capture a SYN packet::

  nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
  tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

Open another terminal, run nc command::

  nstatuser@nstat-a:~$ nc nstat-b 9000

As the nstat-b didn't listen on port 9000, it should reply a RST, and
the nc command exited immediately. It was enough for the tcpdump
command to capture a SYN packet. A linux server might use hardware
offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
might be not correct. We call tcprewrite to fix it::

  nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum

On nstat-b, we run nc to listen on port 9000::

  nstatuser@nstat-b:~$ nc -lkv 9000
  Listening on [0.0.0.0] (family 0, port 9000)

On nstat-a, we blocked the packet from port 9000, or nstat-a would send
RST to nstat-b::

  nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP

Send 3 SYN repeatedly to nstat-b::

  nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done

Check snmp counter on nstat-b::

  nstatuser@nstat-b:~$ nstat | grep -i skip
  TcpExtTCPACKSkippedSynRecv      1                  0.0

As we expected, TcpExtTCPACKSkippedSynRecv is 1.

예제: TCPACKSkippedPAWS

PAWS ACK 생략은 오래된 SYN을 재생해 만듭니다. 정상 연결의 SYN을 tcpdump로 저장하고 checksum을 고친 뒤 같은 SYN을 두 번 `tcpreplay`합니다. 두 packet 모두 PAWS에 실패하지만 server는 첫 번째에는 ACK하고 두 번째에는 rate limit으로 ACK를 생략해 `TcpExtTCPACKSkippedPAWS=1`이 됩니다.

TcpExtTCPACKSkippedPAWS
-----------------------
To trigger PAWS, we could send an old SYN.

On nstat-b, let nc listen on port 9000::

  nstatuser@nstat-b:~$ nc -lkv 9000
  Listening on [0.0.0.0] (family 0, port 9000)

On nstat-a, run tcpdump to capture a SYN::

  nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
  tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

On nstat-a, run nc as a client to connect nstat-b::

  nstatuser@nstat-a:~$ nc -v nstat-b 9000
  Connection to nstat-b 9000 port [tcp/*] succeeded!

Now the tcpdump has captured the SYN and exit. We should fix the
checksum::

  nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum

Send the SYN packet twice::

  nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done

On nstat-b, check the snmp counter::

  nstatuser@nstat-b:~$ nstat | grep -i skip
  TcpExtTCPACKSkippedPAWS         1                  0.0

We sent two SYN via tcpreplay, both of them would let PAWS check
failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
for the second SYN, and updated TcpExtTCPACKSkippedPAWS.

예제: TCPACKSkippedSeq

`TcpExtTCPACKSkippedSeq`를 만들려면 timestamp는 유효해 PAWS를 통과하지만 sequence가 window 밖인 pure ACK가 필요합니다. data가 있는 packet은 Linux가 이 ACK 생략을 피하므로 port 9000과 9001에 두 연결을 만들고 9001 연결의 ACK를 capture합니다.

`ss`로 두 client source port를 확인한 뒤 `tcprewrite`로 capture packet의 destination port 9001을 9000으로, source port도 9000 연결의 source port로 바꾸고 checksum을 수정합니다. 이 ACK는 9000 socket의 timestamp 문맥에는 유효하지만 sequence가 맞지 않습니다. packet을 두 번 재생하면 첫 번째 응답 뒤 두 번째 ACK가 rate limit으로 생략되어 server의 `TcpExtTCPACKSkippedSeq=1`이 됩니다.

TcpExtTCPACKSkippedSeq
----------------------
To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
timestamp (to pass PAWS check) but the sequence number is out of
window. The linux TCP stack would avoid to skip if the packet has
data, so we need a pure ACK packet. To generate such a packet, we
could create two sockets: one on port 9000, another on port 9001. Then
we capture an ACK on port 9001, change the source/destination port
numbers to match the port 9000 socket. Then we could trigger
TcpExtTCPACKSkippedSeq via this packet.

On nstat-b, open two terminals, run two nc commands to listen on both
port 9000 and port 9001::

  nstatuser@nstat-b:~$ nc -lkv 9000
  Listening on [0.0.0.0] (family 0, port 9000)

  nstatuser@nstat-b:~$ nc -lkv 9001
  Listening on [0.0.0.0] (family 0, port 9001)

On nstat-a, run two nc clients::

  nstatuser@nstat-a:~$ nc -v nstat-b 9000
  Connection to nstat-b 9000 port [tcp/*] succeeded!

  nstatuser@nstat-a:~$ nc -v nstat-b 9001
  Connection to nstat-b 9001 port [tcp/*] succeeded!

On nstat-a, run tcpdump to capture an ACK::

  nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
  tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
string 'foo' in our example::

  nstatuser@nstat-b:~$ nc -lkv 9001
  Listening on [0.0.0.0] (family 0, port 9001)
  Connection from nstat-a 42132 received!
  foo

On nstat-a, the tcpdump should have captured the ACK. We should check
the source port numbers of the two nc clients::

  nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
  State  Recv-Q   Send-Q         Local Address:Port           Peer Address:Port
  ESTAB  0        0            192.168.122.250:50208       192.168.122.251:9000
  ESTAB  0        0            192.168.122.250:42132       192.168.122.251:9001

Run tcprewrite, change port 9001 to port 9000, change port 42132 to
port 50208::

  nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum

Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::

  nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done

Check TcpExtTCPACKSkippedSeq on nstat-b::

  nstatuser@nstat-b:~$ nstat | grep -i skip
  TcpExtTCPACKSkippedSeq          1                  0.0