Multipath TCP (MPTCP)

소개와 사용 사례

MPTCP는 RFC 8684(MPTCPv1)에 정의된 standard TCP 확장으로, 하나의 MPTCP connection에서 여러 interface를 동시에 이용해 TCP packet을 송수신합니다. 여러 interface의 bandwidth를 합치거나 latency가 가장 낮은 경로를 선호할 수 있고, 한 경로가 내려가면 traffic을 다른 경로에 매끄럽게 reinject해 failover합니다. Linux 구현에 관한 추가 정보는 `mptcp.dev`에서 제공합니다.

대표 사용 사례는 세 가지입니다. Seamless handover는 smartphone 같은 이동 환경에서 기존 connection을 유지한 채 path를 바꿉니다. Best network selection은 latency, loss, cost, bandwidth 같은 조건에 따라 최적 path를 고릅니다. Network aggregation은 fixed와 mobile network처럼 여러 path를 동시에 사용해 처리량을 높입니다.

.. SPDX-License-Identifier: GPL-2.0

=====================
Multipath TCP (MPTCP)
=====================

Introduction
============

Multipath TCP or MPTCP is an extension to the standard TCP and is described in
`RFC 8684 (MPTCPv1) <https://www.rfc-editor.org/rfc/rfc8684.html>`_. It allows a
device to make use of multiple interfaces at once to send and receive TCP
packets over a single MPTCP connection. MPTCP can aggregate the bandwidth of
multiple interfaces or prefer the one with the lowest latency. It also allows a
fail-over if one path is down, and the traffic is seamlessly reinjected on other
paths.

For more details about Multipath TCP in the Linux kernel, please see the
official website: `mptcp.dev <https://www.mptcp.dev>`_.


Use cases
=========

Thanks to MPTCP, being able to use multiple paths in parallel or simultaneously
brings new use-cases, compared to TCP:

- Seamless handovers: switching from one path to another while preserving
  established connections, e.g. to be used in mobility use-cases, like on
  smartphones.
- Best network selection: using the "best" available path depending on some
  conditions, e.g. latency, losses, cost, bandwidth, etc.
- Network aggregation: using multiple paths at the same time to have a higher
  throughput, e.g. to combine fixed and mobile networks to send files faster.

Subflow와 path manager

Linux 전용 `IPPROTO_MPTCP`로 socket을 만들면 먼저 subflow 또는 path가 하나 생깁니다. Subflow는 한 interface를 통해 data를 전송하는 일반 TCP connection이며 host끼리 추가 subflow를 나중에 협상할 수 있습니다. Underlying TCP subflow의 option에는 MPTCP 지원을 알리는 `MP_CAPABLE` 등이 들어갑니다.

Remote host나 중간 middlebox가 MPTCP를 지원하지 않으면 돌아온 `SYN+ACK`의 TCP option에 MPTCP option이 없습니다. 이 경우 connection은 단일 path의 일반 TCP로 downgrade되어 계속 동작합니다. 이 동작은 path manager와 packet scheduler 두 내부 component로 구현됩니다.

Path manager는 subflow 생성부터 삭제와 address announcement를 담당합니다. 보통 client가 subflow를 시작하고 server가 `ADD_ADDR`, `REMOVE_ADDR` option으로 추가 address를 알립니다. `net.mptcp.path_manager`가 종류를 선택하며, `kernel` manager는 `ip mptcp`로 설정한 동일 rule을 모든 connection에 적용하고 `userspace` manager는 `mptcpd` 같은 daemon이 connection마다 다른 rule을 적용합니다. 둘 다 MPTCP PM Netlink API로 제어할 수 있습니다.

Default in-kernel path manager가 host의 여러 IP address로 subflow를 만들려면 사용할 address를 알아야 합니다. 예를 들어 `ip mptcp endpoint`로 이를 설정합니다.

Concepts
========

Technically, when a new socket is created with the ``IPPROTO_MPTCP`` protocol
(Linux-specific), a *subflow* (or *path*) is created. This *subflow* consists of
a regular TCP connection that is used to transmit data through one interface.
Additional *subflows* can be negotiated later between the hosts. For the remote
host to be able to detect the use of MPTCP, a new field is added to the TCP
*option* field of the underlying TCP *subflow*. This field contains, amongst
other things, a ``MP_CAPABLE`` option that tells the other host to use MPTCP if
it is supported. If the remote host or any middlebox in between does not support
it, the returned ``SYN+ACK`` packet will not contain MPTCP options in the TCP
*option* field. In that case, the connection will be "downgraded" to plain TCP,
and it will continue with a single path.

This behavior is made possible by two internal components: the path manager, and
the packet scheduler.

Path Manager
------------

The Path Manager is in charge of *subflows*, from creation to deletion, and also
address announcements. Typically, it is the client side that initiates subflows,
and the server side that announces additional addresses via the ``ADD_ADDR`` and
``REMOVE_ADDR`` options.

Path managers are controlled by the ``net.mptcp.path_manager`` sysctl knob --
see mptcp-sysctl.rst. There are two types: the in-kernel one (``kernel``) where
the same rules are applied for all the connections (see: ``ip mptcp``) ; and the
userspace one (``userspace``), controlled by a userspace daemon (i.e. `mptcpd
<https://mptcpd.mptcp.dev/>`_) where different rules can be applied for each
connection. The path managers can be controlled via a Netlink API; see
../netlink/specs/mptcp_pm.rst.

To be able to use multiple IP addresses on a host to create multiple *subflows*
(paths), the default in-kernel MPTCP path-manager needs to know which IP
addresses can be used. This can be configured with ``ip mptcp endpoint`` for
example.

Packet scheduler

Packet scheduler는 다음 data packet을 보낼 때 사용 가능한 subflow 하나 또는 여러 개를 선택합니다. Configuration에 따라 전체 bandwidth 이용을 극대화하거나 latency가 가장 낮은 path만 고르는 등 다양한 policy를 구현할 수 있습니다. `net.mptcp.scheduler` sysctl로 scheduler를 선택하며 자세한 설정은 `mptcp-sysctl.rst`에 있습니다.

Packet Scheduler
----------------

The Packet Scheduler is in charge of selecting which available *subflow(s)* to
use to send the next data packet. It can decide to maximize the use of the
available bandwidth, only to pick the path with the lower latency, or any other
policy depending on the configuration.

Packet schedulers are controlled by the ``net.mptcp.scheduler`` sysctl knob --
see mptcp-sysctl.rst.

MPTCP socket 생성

Linux에서는 TCP 대신 `socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP)`를 선택해 MPTCP socket을 만듭니다. `IPPROTO_MPTCP` 값은 262입니다. 지원되지 않으면 Linux 5.6 이전 kernel은 `EINVAL`, 5.6 이상에서 MPTCP를 compile하지 않은 kernel은 `EPROTONOSUPPORT`, `net.mptcp.enabled`로 비활성화한 경우는 `ENOPROTOOPT`를 설정합니다.

MPTCP는 application이 명시적으로 요청하는 opt-in 방식입니다. 다만 `mptcpize`의 `LD_PRELOAD`, `mptcpify`의 eBPF, SystemTap, Go의 `GODEBUG=multipathtcp=1` 같은 방법으로 application에 MPTCP 사용을 강제할 수 있습니다. `IPPROTO_TCP`를 `IPPROTO_MPTCP`로 바꾸는 작업은 userspace application에 가능한 한 투명해야 합니다.

Sockets API
===========

Creating MPTCP sockets
----------------------

On Linux, MPTCP can be used by selecting MPTCP instead of TCP when creating the
``socket``:

.. code-block:: C

    int sd = socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP);

Note that ``IPPROTO_MPTCP`` is defined as ``262``.

If MPTCP is not supported, ``errno`` will be set to:

- ``EINVAL``: (*Invalid argument*): MPTCP is not available, on kernels < 5.6.
- ``EPROTONOSUPPORT`` (*Protocol not supported*): MPTCP has not been compiled,
  on kernels >= v5.6.
- ``ENOPROTOOPT`` (*Protocol not available*): MPTCP has been disabled using
  ``net.mptcp.enabled`` sysctl knob; see mptcp-sysctl.rst.

MPTCP is then opt-in: applications need to explicitly request it. Note that
applications can be forced to use MPTCP with different techniques, e.g.
``LD_PRELOAD`` (see ``mptcpize``), eBPF (see ``mptcpify``), SystemTAP,
``GODEBUG`` (``GODEBUG=multipathtcp=1``), etc.

Switching to ``IPPROTO_MPTCP`` instead of ``IPPROTO_TCP`` should be as
transparent as possible for the userspace applications.

Socket option

MPTCP는 TCP가 처리하는 socket option 대부분을 지원합니다. 드문 option 일부는 아직 지원하지 않을 수 있습니다. 일반적으로 `setsockopt()`로 설정한 값은 호출 뒤 새로 생긴 subflow까지 포함해 모든 subflow에 전파되며, subflow별로 다른 값이 필요하면 eBPF를 사용할 수 있습니다.

`SOL_MPTCP` level 284에는 `getsockopt()`의 `optval` buffer로 정보를 돌려주는 MPTCP 전용 option이 있습니다. `MPTCP_INFO`는 `struct mptcp_info`, `MPTCP_TCPINFO`는 `struct mptcp_subflow_data` 뒤의 `tcp_info` 배열, `MPTCP_SUBFLOW_ADDRS`는 같은 header 뒤의 `mptcp_subflow_addrs` 배열을 사용합니다.

`MPTCP_FULL_INFO`는 `mptcp_subflow_info` 배열과 `tcp_info` 배열을 가리키는 pointer 및 `mptcp_info` 내용을 함께 담습니다. TCP level의 `TCP_IS_MPTCP` option은 현재 MPTCP를 사용 중이면 1이므로 connection이 실제 MPTCP인지 확인할 수 있습니다.

Socket options
--------------

MPTCP supports most socket options handled by TCP. It is possible some less
common options are not supported, but contributions are welcome.

Generally, the same value is propagated to all subflows, including the ones
created after the calls to ``setsockopt()``. eBPF can be used to set different
values per subflow.

There are some MPTCP specific socket options at the ``SOL_MPTCP`` (284) level to
retrieve info. They fill the ``optval`` buffer of the ``getsockopt()`` system
call:

- ``MPTCP_INFO``: Uses ``struct mptcp_info``.
- ``MPTCP_TCPINFO``: Uses ``struct mptcp_subflow_data``, followed by an array of
  ``struct tcp_info``.
- ``MPTCP_SUBFLOW_ADDRS``: Uses ``struct mptcp_subflow_data``, followed by an
  array of ``mptcp_subflow_addrs``.
- ``MPTCP_FULL_INFO``: Uses ``struct mptcp_full_info``, with one pointer to an
  array of ``struct mptcp_subflow_info`` (including the
  ``struct mptcp_subflow_addrs``), and one pointer to an array of
  ``struct tcp_info``, followed by the content of ``struct mptcp_info``.

Note that at the TCP level, ``TCP_IS_MPTCP`` socket option can be used to know
if MPTCP is currently being used: the value will be set to 1 if it is.

Kernel 설계 선택

Userspace가 보는 MPTCP socket을 위해 새 socket type을 추가했습니다. Kernel은 subflow socket을 만들며, 이들은 TCP-ULP로 동작을 수정한 TCP socket입니다.

MPTCP listen socket이 MPTCP를 요청하지 않은 client의 connection request를 받으면 일반 accepted TCP socket을 만듭니다. 따라서 MPTCP를 기본 활성화해도 plain TCP connection의 성능 영향은 최소화됩니다.

Design choices
==============

A new socket type has been added for MPTCP for the userspace-facing socket. The
kernel is in charge of creating subflow sockets: they are TCP sockets where the
behavior is modified using TCP-ULP.

MPTCP listen sockets will create "plain" *accepted* TCP sockets if the
connection request from the client didn't ask for MPTCP, making the performance
impact minimal when MPTCP is enabled by default.