Network timestamping

Timestamp control interface 개요

이 문서는 GPL-2.0 라이선스를 따르며 network packet timestamp를 생성하고 읽는 Linux interface를 설명합니다.

`SO_TIMESTAMP`는 incoming packet마다 반드시 monotonic하지는 않은 system time timestamp를 만들고 `recvmsg()` control message로 microsecond 해상도에서 보고합니다. Architecture와 libc `time_t` 표현에 따라 `SO_TIMESTAMP_NEW` 또는 `SO_TIMESTAMP_OLD`로 정의됩니다. OLD는 `struct __kernel_old_timeval`, NEW는 `struct __kernel_sock_timeval` 형식을 사용합니다.

`SO_TIMESTAMPNS`는 같은 mechanism을 nanosecond 해상도의 `struct timespec`으로 보고합니다. OLD는 `struct timespec`, NEW는 `struct __kernel_timespec` 형식입니다.

Multicast에서는 `IP_MULTICAST_LOOP`와 `SO_TIMESTAMP[NS]`를 함께 사용해 loopback으로 돌아온 packet의 receive timestamp를 읽는 방식으로 근사 transmit timestamp를 얻을 수 있습니다.

`SO_TIMESTAMPING`은 receive, transmit 또는 양쪽 timestamp를 생성하며 hardware를 포함한 여러 source와 stream socket timestamping을 지원합니다.

.. SPDX-License-Identifier: GPL-2.0

============
Timestamping
============


1. Control Interfaces
=====================

The interfaces for receiving network packages timestamps are:

SO_TIMESTAMP
  Generates a timestamp for each incoming packet in (not necessarily
  monotonic) system time. Reports the timestamp via recvmsg() in a
  control message in usec resolution.
  SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD
  based on the architecture type and time_t representation of libc.
  Control message format is in struct __kernel_old_timeval for
  SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for
  SO_TIMESTAMP_NEW options respectively.

SO_TIMESTAMPNS
  Same timestamping mechanism as SO_TIMESTAMP, but reports the
  timestamp as struct timespec in nsec resolution.
  SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD
  based on the architecture type and time_t representation of libc.
  Control message format is in struct timespec for SO_TIMESTAMPNS_OLD
  and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options
  respectively.

IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
  Only for multicast:approximate transmit timestamp obtained by
  reading the looped packet receive timestamp.

SO_TIMESTAMPING
  Generates timestamps on reception, transmission or both. Supports
  multiple timestamp sources, including hardware. Supports generating
  timestamps for stream sockets.

SO_TIMESTAMP와 SO_TIMESTAMPNS

`SO_TIMESTAMP`는 receive path의 datagram timestamping을 활성화합니다. Network stack 초반에는 destination socket을 아직 알 수 없으므로 모든 packet에 대해 기능을 켜야 하며, 다른 early receive timestamp option도 같습니다. 자세한 interface는 `man 7 socket`을 참조합니다.

항상 `struct __kernel_sock_timeval`을 받으려면 `SO_TIMESTAMP_NEW`를 사용해야 합니다. 32-bit machine에서 `SO_TIMESTAMP_OLD`는 2038년 이후 잘못된 timestamp를 반환합니다.

`SO_TIMESTAMPNS`는 반환 data type을 제외하면 `SO_TIMESTAMP`와 동일하며, `timeval`보다 높은 nanosecond 해상도의 `struct timespec`을 사용합니다. 항상 `struct __kernel_timespec`을 받으려면 `SO_TIMESTAMPNS_NEW`를 사용해야 합니다. 32-bit machine의 OLD variant는 역시 2038년 이후 잘못됩니다.

1.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW)
-------------------------------------------------------------

This socket option enables timestamping of datagrams on the reception
path. Because the destination socket, if any, is not known early in
the network stack, the feature has to be enabled for all packets. The
same is true for all early receive timestamp options.

For interface details, see `man 7 socket`.

Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in
struct __kernel_sock_timeval format.

SO_TIMESTAMP_OLD returns incorrect timestamps after the year 2038
on 32 bit machines.

1.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW)
-------------------------------------------------------------------

This option is identical to SO_TIMESTAMP except for the returned data type.
Its struct timespec allows for higher resolution (ns) timestamps than the
timeval of SO_TIMESTAMP (ms).

Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in
struct __kernel_timespec format.

SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 2038
on 32 bit machines.

SO_TIMESTAMPING generation flag

`SO_TIMESTAMPING`은 여러 timestamp request type을 지원하므로 boolean이 아니라 flag bitmap을 받습니다. `setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val))`로 설정하며 정의되지 않은 bit가 있으면 `EINVAL`을 반환하고 현재 상태를 바꾸지 않습니다.

Socket option은 개별 `sk_buff`의 timestamp generation, socket error queue 보고, 추가 option을 함께 설정합니다. 개별 `sendmsg()` 호출은 cmsg로 generation만 따로 요청할 수도 있습니다.

Generation bit는 stack에 timestamp 생성을 시도하라고 요청합니다. 조합은 자유롭고 변경은 새 packet에만 적용됩니다. 따라서 enable과 disable `setsockopt()` 사이에 `send()`를 두어 일부 packet만 sampling할 수 있습니다. System-wide RX timestamping처럼 특정 socket request 이외의 이유로도 timestamp가 생성될 수 있습니다.

`TX_SCHED`와 `TX_SOFTWARE`의 차이는 scheduler queuing delay를 보여 줍니다. `send()` 직전 userspace timestamp와 `TX_SCHED`의 차이는 protocol processing latency를 나타냅니다.

`TX_ACK`는 `send()` buffer까지 포함한 모든 data가 cumulative ACK됐을 때 생성되어 측정치를 크게 보고할 수 있습니다. `TX_COMPLETION`은 hardware가 여러 packet을 묶어 완료 보고할 수 있으므로 실제 wire transmit time이 아니라 completion report timing입니다.

1.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW)
----------------------------------------------------------------------

Supports multiple types of timestamp requests. As a result, this
socket option takes a bitmap of flags, not a boolean. In::

  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));

val is an integer with any of the following bits set. Setting other
bit returns EINVAL and does not change the current state.

The socket option configures timestamp generation for individual
sk_buffs (1.3.1), timestamp reporting to the socket's error
queue (1.3.2) and options (1.3.3). Timestamp generation can also
be enabled for individual sendmsg calls using cmsg (1.3.4).


1.3.1 Timestamp Generation
^^^^^^^^^^^^^^^^^^^^^^^^^^

Some bits are requests to the stack to try to generate timestamps. Any
combination of them is valid. Changes to these bits apply to newly
created packets, not to packets already in the stack. As a result, it
is possible to selectively request timestamps for a subset of packets
(e.g., for sampling) by embedding an send() call within two setsockopt
calls, one to enable timestamp generation and one to disable it.
Timestamps may also be generated for reasons other than being
requested by a particular socket, such as when receive timestamping is
enabled system wide, as explained earlier.

SOF_TIMESTAMPING_RX_HARDWARE:
  Request rx timestamps generated by the network adapter.

SOF_TIMESTAMPING_RX_SOFTWARE:
  Request rx timestamps when data enters the kernel. These timestamps
  are generated just after a device driver hands a packet to the
  kernel receive stack.

SOF_TIMESTAMPING_TX_HARDWARE:
  Request tx timestamps generated by the network adapter. This flag
  can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SOFTWARE:
  Request tx timestamps when data leaves the kernel. These timestamps
  are generated in the device driver as close as possible, but always
  prior to, passing the packet to the network interface. Hence, they
  require driver support and may not be available for all devices.
  This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SCHED:
  Request tx timestamps prior to entering the packet scheduler. Kernel
  transmit latency is, if long, often dominated by queuing delay. The
  difference between this timestamp and one taken at
  SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
  of protocol processing. The latency incurred in protocol
  processing, if any, can be computed by subtracting a userspace
  timestamp taken immediately before send() from this timestamp. On
  machines with virtual devices where a transmitted packet travels
  through multiple devices and, hence, multiple packet schedulers,
  a timestamp is generated at each layer. This allows for fine
  grained measurement of queuing delay. This flag can be enabled
  via both socket options and control messages.

SOF_TIMESTAMPING_TX_ACK:
  Request tx timestamps when all data in the send buffer has been
  acknowledged. This only makes sense for reliable protocols. It is
  currently only implemented for TCP. For that protocol, it may
  over-report measurement, because the timestamp is generated when all
  data up to and including the buffer at send() was acknowledged: the
  cumulative acknowledgment. The mechanism ignores SACK and FACK.
  This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_COMPLETION:
  Request tx timestamps on packet tx completion.  The completion
  timestamp is generated by the kernel when it receives packet a
  completion report from the hardware. Hardware may report multiple
  packets at once, and completion timestamps reflect the timing of the
  report and not actual tx time. This flag can be enabled via both
  socket options and control messages.

Reporting flag와 timestamp option

Reporting bit는 생성된 control message에 어떤 timestamp를 실을지 결정하며 stack의 보고 지점에 즉시 적용됩니다. 해당 generation request가 설정된 packet의 timestamp만 보고됩니다.

`SOF_TIMESTAMPING_SOFTWARE`는 가능한 software timestamp를 보고합니다. `SOF_TIMESTAMPING_SYS_HARDWARE`는 deprecated되어 무시됩니다. `SOF_TIMESTAMPING_RAW_HARDWARE`는 TX/RX hardware generation flag가 만든 raw hardware timestamp를 보고합니다.

`SOF_TIMESTAMPING_OPT_ID`는 concurrent request와 TX path reordering에도 원래 `send()`를 식별할 수 있도록 packet마다 unique ID를 붙입니다. Socket별 wrapping `u32` counter에서 만들며 datagram은 packet마다, stream은 byte마다 증가합니다. Stream socket에서는 `SOF_TIMESTAMPING_OPT_ID_TCP`도 함께 설정해야 합니다.

Counter는 option을 처음 enable할 때 zero로 시작하고 disable 후 다시 enable할 때 reset됩니다. 이미 system 안에 있는 packet의 ID는 바뀌지 않습니다. TX timestamp에만 구현되며 `struct sock_extended_err.ee_data`로 ID를 돌려줍니다. Non-TCP socket은 `SCM_TS_OPT_ID` cmsg로 application이 ID를 덮어쓸 수도 있습니다. 원문의 `msghdr`/`cmsghdr` 예제를 그대로 보존합니다.

`SOF_TIMESTAMPING_OPT_ID_TCP`는 stream ID 시작점을 안정적으로 정의합니다. 이후 N byte `write()`가 counter `N-1` timestamp를 만들도록 항상 syscall 시점에 zero로 맞춥니다. Modifier 없는 방식은 output queue가 있으면 `SIOCOUTQ` 길이만큼 어긋날 수 있습니다. 이는 peer ACK와 RTT에 좌우되는 `snd_una` 대신 process가 마지막으로 쓴 byte인 `write_seq`를 기준으로 삼는 차이입니다.

`SOF_TIMESTAMPING_OPT_CMSG`는 RX와 IPv6 TX에 이미 제공되는 cmsg를 IPv4 TX까지 확장하며 `IP_PKTINFO`와 함께 egress interface를 연계할 수 있습니다.

`SOF_TIMESTAMPING_OPT_TSONLY`는 TX timestamp를 original packet 대신 empty packet 옆 cmsg로 반환해 `SO_RCVBUF` 부담을 줄이고 `net.core.tstamp_allow_data=0`에서도 전달합니다. 이 option은 `OPT_CMSG`를 끕니다. `OPT_STATS`는 `OPT_TSONLY`와 함께 사용하며 transport statistic을 `SCM_TIMESTAMPING_OPT_STATS`의 `nlattr` TLV list로 전달합니다.

`OPT_PKTINFO`는 hardware timestamp가 있는 incoming packet에 `SCM_TIMESTAMPING_PKTINFO`를 추가합니다. `struct scm_ts_pktinfo`에는 실제 receive interface index와 L2 length가 있고, non-zero index는 `CONFIG_NET_RX_BUSY_POLL`과 NAPI driver가 모두 필요합니다.

`OPT_TX_SWHW`는 TX hardware와 software timestamp가 모두 생성되면 각각 하나의 timestamp만 담은 두 message를 error queue에 넣습니다. `OPT_RX_FILTER`는 ingress 초반의 system-wide generation 때문에 TX-only socket까지 받게 되는 spurious RX timestamp를 제거해 다른 socket과 독립적인 동작을 만듭니다.

새 application은 timestamp 식별을 위해 `OPT_ID`, data 허용 sysctl과 무관한 동작을 위해 `OPT_TSONLY`를 권장합니다. Egress interface용 `IP_PKTINFO` 같은 추가 cmsg가 필요하면 original packet content에 의존하는 `OPT_CMSG`를 사용해야 하므로 `OPT_TSONLY`와 함께 쓸 수 없습니다.

1.3.2 Timestamp Reporting
^^^^^^^^^^^^^^^^^^^^^^^^^

The other three bits control which timestamps will be reported in a
generated control message. Changes to the bits take immediate
effect at the timestamp reporting locations in the stack. Timestamps
are only reported for packets that also have the relevant timestamp
generation request set.

SOF_TIMESTAMPING_SOFTWARE:
  Report any software timestamps when available.

SOF_TIMESTAMPING_SYS_HARDWARE:
  This option is deprecated and ignored.

SOF_TIMESTAMPING_RAW_HARDWARE:
  Report hardware timestamps as generated by
  SOF_TIMESTAMPING_TX_HARDWARE or SOF_TIMESTAMPING_RX_HARDWARE
  when available.


1.3.3 Timestamp Options
^^^^^^^^^^^^^^^^^^^^^^^

The interface supports the options

SOF_TIMESTAMPING_OPT_ID:
  Generate a unique identifier along with each packet. A process can
  have multiple concurrent timestamping requests outstanding. Packets
  can be reordered in the transmit path, for instance in the packet
  scheduler. In that case timestamps will be queued onto the error
  queue out of order from the original send() calls. It is not always
  possible to uniquely match timestamps to the original send() calls
  based on timestamp order or payload inspection alone, then.

  This option associates each packet at send() with a unique
  identifier and returns that along with the timestamp. The identifier
  is derived from a per-socket u32 counter (that wraps). For datagram
  sockets, the counter increments with each sent packet. For stream
  sockets, it increments with every byte. For stream sockets, also set
  SOF_TIMESTAMPING_OPT_ID_TCP, see the section below.

  The counter starts at zero. It is initialized the first time that
  the socket option is enabled. It is reset each time the option is
  enabled after having been disabled. Resetting the counter does not
  change the identifiers of existing packets in the system.

  This option is implemented only for transmit timestamps. There, the
  timestamp is always looped along with a struct sock_extended_err.
  The option modifies field ee_data to pass an id that is unique
  among all possibly concurrently outstanding timestamp requests for
  that socket.

  The process can optionally override the default generated ID, by
  passing a specific ID with control message SCM_TS_OPT_ID (not
  supported for TCP sockets)::

    struct msghdr *msg;
    ...
    cmsg                         = CMSG_FIRSTHDR(msg);
    cmsg->cmsg_level                 = SOL_SOCKET;
    cmsg->cmsg_type                 = SCM_TS_OPT_ID;
    cmsg->cmsg_len                 = CMSG_LEN(sizeof(__u32));
    *((__u32 *) CMSG_DATA(cmsg)) = opt_id;
    err = sendmsg(fd, msg, 0);


SOF_TIMESTAMPING_OPT_ID_TCP:
  Pass this modifier along with SOF_TIMESTAMPING_OPT_ID for new TCP
  timestamping applications. SOF_TIMESTAMPING_OPT_ID defines how the
  counter increments for stream sockets, but its starting point is
  not entirely trivial. This option fixes that.

  For stream sockets, if SOF_TIMESTAMPING_OPT_ID is set, this should
  always be set too. On datagram sockets the option has no effect.

  A reasonable expectation is that the counter is reset to zero with
  the system call, so that a subsequent write() of N bytes generates
  a timestamp with counter N-1. SOF_TIMESTAMPING_OPT_ID_TCP
  implements this behavior under all conditions.

  SOF_TIMESTAMPING_OPT_ID without modifier often reports the same,
  especially when the socket option is set when no data is in
  transmission. If data is being transmitted, it may be off by the
  length of the output queue (SIOCOUTQ).

  The difference is due to being based on snd_una versus write_seq.
  snd_una is the offset in the stream acknowledged by the peer. This
  depends on factors outside of process control, such as network RTT.
  write_seq is the last byte written by the process. This offset is
  not affected by external inputs.

  The difference is subtle and unlikely to be noticed when configured
  at initial socket creation, when no data is queued or sent. But
  SOF_TIMESTAMPING_OPT_ID_TCP behavior is more robust regardless of
  when the socket option is set.

SOF_TIMESTAMPING_OPT_CMSG:
  Support recv() cmsg for all timestamped packets. Control messages
  are already supported unconditionally on all packets with receive
  timestamps and on IPv6 packets with transmit timestamp. This option
  extends them to IPv4 packets with transmit timestamp. One use case
  is to correlate packets with their egress device, by enabling socket
  option IP_PKTINFO simultaneously.


SOF_TIMESTAMPING_OPT_TSONLY:
  Applies to transmit timestamps only. Makes the kernel return the
  timestamp as a cmsg alongside an empty packet, as opposed to
  alongside the original packet. This reduces the amount of memory
  charged to the socket's receive budget (SO_RCVBUF) and delivers
  the timestamp even if sysctl net.core.tstamp_allow_data is 0.
  This option disables SOF_TIMESTAMPING_OPT_CMSG.

SOF_TIMESTAMPING_OPT_STATS:
  Optional stats that are obtained along with the transmit timestamps.
  It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the
  transmit timestamp is available, the stats are available in a
  separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a
  list of TLVs (struct nlattr) of types. These stats allow the
  application to associate various transport layer stats with
  the transmit timestamps, such as how long a certain block of
  data was limited by peer's receiver window.

SOF_TIMESTAMPING_OPT_PKTINFO:
  Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming
  packets with hardware timestamps. The message contains struct
  scm_ts_pktinfo, which supplies the index of the real interface which
  received the packet and its length at layer 2. A valid (non-zero)
  interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is
  enabled and the driver is using NAPI. The struct contains also two
  other fields, but they are reserved and undefined.

SOF_TIMESTAMPING_OPT_TX_SWHW:
  Request both hardware and software timestamps for outgoing packets
  when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE
  are enabled at the same time. If both timestamps are generated,
  two separate messages will be looped to the socket's error queue,
  each containing just one timestamp.

SOF_TIMESTAMPING_OPT_RX_FILTER:
  Filter out spurious receive timestamps: report a receive timestamp
  only if the matching timestamp generation flag is enabled.

  Receive timestamps are generated early in the ingress path, before a
  packet's destination socket is known. If any socket enables receive
  timestamps, packets for all socket will receive timestamped packets.
  Including those that request timestamp reporting with
  SOF_TIMESTAMPING_SOFTWARE and/or SOF_TIMESTAMPING_RAW_HARDWARE, but
  do not request receive timestamp generation. This can happen when
  requesting transmit timestamps only.

  Receiving spurious timestamps is generally benign. A process can
  ignore the unexpected non-zero value. But it makes behavior subtly
  dependent on other sockets. This flag isolates the socket for more
  deterministic behavior.

New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
regardless of the setting of sysctl net.core.tstamp_allow_data.

An exception is when a process needs additional cmsg data, for
instance SOL_IP/IP_PKTINFO to detect the egress network interface.
Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
having access to the contents of the original packet, so cannot be
combined with SOF_TIMESTAMPING_OPT_TSONLY.

Send별 cmsg generation 요청

Socket option 외에도 `SOF_TIMESTAMPING_TX_*` generation flag는 write별 cmsg로 요청할 수 있습니다. `setsockopt()`를 반복해 enable/disable하는 비용 없이 `sendmsg()`별 timestamp sampling이 가능합니다. 원문의 `msghdr`와 flag 예제를 그대로 보존합니다.

Cmsg로 설정한 `SOF_TIMESTAMPING_TX_*` flag는 `setsockopt()`의 TX generation flag를 덮어씁니다. 다만 timestamp를 실제로 받으려면 `SOF_TIMESTAMPING_SOFTWARE`, `SOF_TIMESTAMPING_OPT_ID` 같은 reporting/option flag를 `setsockopt()`로 계속 enable해야 합니다.

1.3.4. Enabling timestamps via control messages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In addition to socket options, timestamp generation can be requested
per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).
Using this feature, applications can sample timestamps per sendmsg()
without paying the overhead of enabling and disabling timestamps via
setsockopt::

  struct msghdr *msg;
  ...
  cmsg                               = CMSG_FIRSTHDR(msg);
  cmsg->cmsg_level               = SOL_SOCKET;
  cmsg->cmsg_type               = SO_TIMESTAMPING;
  cmsg->cmsg_len               = CMSG_LEN(sizeof(__u32));
  *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |
                                 SOF_TIMESTAMPING_TX_SOFTWARE |
                                 SOF_TIMESTAMPING_TX_ACK;
  err = sendmsg(fd, msg, 0);

The SOF_TIMESTAMPING_TX_* flags set via cmsg will override
the SOF_TIMESTAMPING_TX_* flags set via setsockopt.

Moreover, applications must still enable timestamp reporting via
setsockopt to receive timestamps::

  __u32 val = SOF_TIMESTAMPING_SOFTWARE |
              SOF_TIMESTAMPING_OPT_ID /* or any other flag */;
  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));

Bytestream timestamp 의미

`SO_TIMESTAMPING`은 bytestream의 byte timestamping을 지원합니다. Request는 buffer 전체 내용이 timestamp point를 지난 시각을 요구하는 것으로 해석합니다. 예를 들어 stream의 `TX_SOFTWARE`는 data가 몇 packet으로 바뀌었는지와 무관하게 모든 byte가 device driver에 도달했을 때 기록합니다.

Bytestream에는 자연스러운 delimiter가 없어 timestamp와 data 연계가 어렵습니다. Byte range는 segment로 분할되거나 합쳐질 수 있고 서로 다른 `send()` buffer 일부가 coalesce될 수 있습니다. Segment는 reorder될 수 있고 retransmission protocol에서는 같은 byte range가 여러 segment에 공존합니다.

이 변환과 관계없이 모든 timestamp가 같은 의미를 구현해야 비교할 수 있습니다. 성능 debugging은 드문 outlier를 다루므로 buffer와 skb가 1:1인 단순 경우와 corner case를 다르게 처리하면 안 됩니다.

Linux는 모든 byte가 point를 지날 때만 timestamp를 생성합니다. IP fragmentation이 첫 fragment만 timestamp하는 정책과 같은 명시적 의미 선택입니다. `TX_ACK`는 구현과 추론이 쉽지만 SACK까지 고려하면 hole과 out-of-order 때문에 복잡해집니다.

TCP의 Nagle, cork, autocork, segmentation과 GSO도 buffer와 skb의 1:1 관계를 깨뜨립니다. 구현은 skb extend/merge 뒤에도 각 `send()`가 넘긴 마지막 byte의 sequence를 `skb_shinfo(skb)->tskey`에 저장합니다. Skb에는 field가 하나뿐이라 timestamp도 하나만 생성할 수 있습니다.

두 request가 같은 skb로 합쳐지면 드물게 timestamp 하나를 놓칠 수 있습니다. `OPT_ID`로 send byte offset과 반환 ID를 비교해 감지합니다. `TCP_NODELAY`를 켜고 `TCP_CORK`와 autocork를 끄거나 Linux 4.7 이후 `sendmsg(MSG_EOR)`로 coalescing을 막을 수 있습니다.

이 조치는 network stack이 stream segment를 reorder하지 않는다는 전제에서 모든 byte가 point를 지난 뒤 timestamp가 생기도록 합니다. Stack은 reordering을 피하지만 관리자가 같은 stream segment에 서로 다른 지연을 주는 packet scheduler를 구성한 예외는 가능합니다.

1.4 Bytestream Timestamps
-------------------------

The SO_TIMESTAMPING interface supports timestamping of bytes in a
bytestream. Each request is interpreted as a request for when the
entire contents of the buffer has passed a timestamping point. That
is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
when all bytes have reached the device driver, regardless of how
many packets the data has been converted into.

In general, bytestreams have no natural delimiters and therefore
correlating a timestamp with data is non-trivial. A range of bytes
may be split across segments, any segments may be merged (possibly
coalescing sections of previously segmented buffers associated with
independent send() calls). Segments can be reordered and the same
byte range can coexist in multiple segments for protocols that
implement retransmissions.

It is essential that all timestamps implement the same semantics,
regardless of these possible transformations, as otherwise they are
incomparable. Handling "rare" corner cases differently from the
simple case (a 1:1 mapping from buffer to skb) is insufficient
because performance debugging often needs to focus on such outliers.

In practice, timestamps can be correlated with segments of a
bytestream consistently, if both semantics of the timestamp and the
timing of measurement are chosen correctly. This challenge is no
different from deciding on a strategy for IP fragmentation. There, the
definition is that only the first fragment is timestamped. For
bytestreams, we chose that a timestamp is generated only when all
bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
implement and reason about. An implementation that has to take into
account SACK would be more complex due to possible transmission holes
and out of order arrival.

On the host, TCP can also break the simple 1:1 mapping from buffer to
skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
implementation ensures correctness in all cases by tracking the
individual last byte passed to send(), even if it is no longer the
last byte after an skbuff extend or merge operation. It stores the
relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
has only one such field, only one timestamp can be generated.

In rare cases, a timestamp request can be missed if two requests are
collapsed onto the same skb. A process can detect this situation by
enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
send time with the value returned for each timestamp. It can prevent
the situation by always flushing the TCP stack in between requests,
for instance by enabling TCP_NODELAY and disabling TCP_CORK and
autocork. After linux-4.7, a better way to prevent coalescing is
to use MSG_EOR flag at sendmsg() time.

These precautions ensure that the timestamp is generated only when all
bytes have passed a timestamp point, assuming that the network stack
itself does not reorder the segments. The stack indeed tries to avoid
reordering. The one exception is under administrator control: it is
possible to construct a packet scheduler configuration that delays
segments from the same stream differently. Such a setup would be
unusual.

Ancillary data와 SCM_TIMESTAMPING record

Timestamp는 `recvmsg()` ancillary data로 읽습니다. Interface는 `man 3 cmsg`, `SO_TIMESTAMP`와 `SO_TIMESTAMPNS` record 회수는 `man 7 socket`을 참조합니다.

`SCM_TIMESTAMPING` control message는 `cmsg_level=SOL_SOCKET`, `cmsg_type=SCM_TIMESTAMPING`입니다. OLD payload는 세 `timespec`을 가진 `struct scm_timestamping`, NEW payload는 세 `__kernel_timespec`을 가진 `struct scm_timestamping64`입니다. 항상 NEW를 사용해야 하며 32-bit machine에서 OLD는 2038년 이후 잘못됩니다.

구조체는 legacy 이유로 최대 세 timestamp를 반환하지만 항상 하나 이상만 non-zero입니다. 대부분 `ts[0]`, hardware timestamp는 `ts[2]`에 있습니다. `ts[1]`의 system-time 변환 hardware timestamp는 deprecated됐습니다. 대신 NIC의 hardware clock을 PTP clock source로 직접 공개해 userspace가 변환하거나 linuxptp로 system time을 동기화합니다.

`SO_TIMESTAMP` 또는 `SO_TIMESTAMPNS`를 `SOF_TIMESTAMPING_SOFTWARE`와 함께 켜면 실제 software timestamp가 없을 때 `recvmsg()`가 거짓 software timestamp를 만들어 `ts[0]`에 넣을 수 있으며 hardware TX timestamp에서도 발생합니다.

2 Data Interfaces
==================

Timestamps are read using the ancillary data feature of recvmsg().
See `man 3 cmsg` for details of this interface. The socket manual
page (`man 7 socket`) describes how timestamps generated with
SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.


2.1 SCM_TIMESTAMPING records
----------------------------

These timestamps are returned in a control message with cmsg_level
SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type

For SO_TIMESTAMPING_OLD::

        struct scm_timestamping {
                struct timespec ts[3];
        };

For SO_TIMESTAMPING_NEW::

        struct scm_timestamping64 {
                struct __kernel_timespec ts[3];

Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in
struct scm_timestamping64 format.

SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038
on 32 bit machines.

The structure can return up to three timestamps. This is a legacy
feature. At least one field is non-zero at any time. Most timestamps
are passed in ts[0]. Hardware timestamps are passed in ts[2].

ts[1] used to hold hardware timestamps converted to system time.
Instead, expose the hardware clock device on the NIC directly as
a HW PTP clock source, to allow time conversion in userspace and
optionally synchronize system time with a userspace PTP stack such
as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.

Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled
together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false
software timestamp will be generated in the recvmsg() call and passed
in ts[0] when a real software timestamp is missing. This happens also
on hardware transmit timestamps.

TX error queue와 RX recvmsg

TX timestamp는 outgoing packet을 send timestamp와 함께 socket error queue로 loopback합니다. Process는 충분한 `msg_control` buffer와 `MSG_ERRQUEUE` flag로 `recvmsg()`를 호출합니다. Original outgoing packet과 ancillary message 두 개가 반환됩니다.

첫 message는 `SOL_IP(V6)`/`IP(V6)_RECVERR`의 `struct sock_extended_err`이며 timestamp에서는 `ee_errno=ENOMSG`입니다. 다른 message는 `SOL_SOCKET`/`SCM_TIMESTAMPING`의 `struct scm_timestamping`입니다.

Extended error의 `ee_info`에 있는 `SCM_TSTAMP_*` 값이 실제 timestamp type을 정의합니다. 대부분 `SOF_TIMESTAMPING_*` control field와 1:1입니다. Legacy `SCM_TSTAMP_SND=0`은 TX hardware와 software 모두에 쓰이며 `ts[2]`가 non-zero면 hardware, 아니면 `ts[0]`의 software timestamp입니다.

Outgoing datagram이 fragmentation되면 첫 fragment만 timestamp되고 sending socket에 반환됩니다.

Application이 original payload를 원하지 않으면 error queue에서 더 작은 buffer나 길이 0으로 읽어 payload를 truncate할 수 있습니다. 그러나 `recvmsg()` 전까지 full packet이 queue되어 `SO_RCVBUF` budget을 사용합니다.

Error queue read는 항상 non-blocking입니다. Timestamp를 기다리며 block하려면 `poll()` 또는 `select()`를 사용합니다. Data가 준비되면 `pollfd.revents`에 `POLLERR`가 오며 `pollfd.events`에 요청할 필요가 없습니다.

RX timestamp는 error queue를 읽지 않습니다. 일반 `recvmsg()`의 packet data와 함께 `SCM_TIMESTAMPING` ancillary data가 오며 socket error가 아니므로 `IP(V6)_RECVERROR` message가 없습니다. `ts[0]`은 software, `ts[1]`은 deprecated, `ts[2]`는 hardware timestamp입니다.

2.1.1 Transmit timestamps with MSG_ERRQUEUE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For transmit timestamps the outgoing packet is looped back to the
socket's error queue with the send timestamp(s) attached. A process
receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
set and with a msg_control buffer sufficiently large to receive the
relevant metadata structures. The recvmsg call returns the original
outgoing data packet with two ancillary messages attached.

A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
embeds a struct sock_extended_err. This defines the error type. For
timestamps, the ee_errno field is ENOMSG. The other ancillary message
will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
embeds the struct scm_timestamping.


2.1.1.2 Timestamp types
~~~~~~~~~~~~~~~~~~~~~~~

The semantics of the three struct timespec are defined by field
ee_info in the extended error structure. It contains a value of
type SCM_TSTAMP_* to define the actual timestamp passed in
scm_timestamping.

The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
control fields discussed previously, with one exception. For legacy
reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
is the first if ts[2] is non-zero, the second otherwise, in which
case the timestamp is stored in ts[0].


2.1.1.3 Fragmentation
~~~~~~~~~~~~~~~~~~~~~

Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
then only the first fragment is timestamped and returned to the sending
socket.


2.1.1.4 Packet Payload
~~~~~~~~~~~~~~~~~~~~~~

The calling application is often not interested in receiving the whole
packet payload that it passed to the stack originally: the socket
error queue mechanism is just a method to piggyback the timestamp on.
In this case, the application can choose to read datagrams with a
smaller buffer, possibly even of length 0. The payload is truncated
accordingly. Until the process calls recvmsg() on the error queue,
however, the full packet is queued, taking up budget from SO_RCVBUF.


2.1.1.5 Blocking Read
~~~~~~~~~~~~~~~~~~~~~

Reading from the error queue is always a non-blocking operation. To
block waiting on a timestamp, use poll or select. poll() will return
POLLERR in pollfd.revents if any data is ready on the error queue.
There is no need to pass this flag in pollfd.events. This flag is
ignored on request. See also `man 2 poll`.


2.1.2 Receive timestamps
^^^^^^^^^^^^^^^^^^^^^^^^

On reception, there is no reason to read from the socket error queue.
The SCM_TIMESTAMPING ancillary data is sent along with the packet data
on a normal recvmsg(). Since this is not a socket error, it is not
accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
the meaning of the three fields in struct scm_timestamping is
implicitly defined. ts[0] holds a software timestamp if set, ts[1]
is again deprecated and ts[2] holds a hardware timestamp if set.

ETHTOOL_MSG_TSCONFIG hardware 설정

Hardware timestamp를 수행할 device driver마다 별도 초기화가 필요합니다. `include/uapi/linux/net_tstamp.h`의 `struct hwtstamp_config`는 현재 반드시 zero인 `flags`, `HWTSTAMP_TX_*`의 `tx_type`, `HWTSTAMP_FILTER_*`의 `rx_filter`를 가집니다.

원하는 동작은 tsconfig netlink `ETHTOOL_MSG_TSCONFIG_SET`으로 특정 device에 전달합니다. `ETHTOOL_A_TSCONFIG_TX_TYPES`, `ETHTOOL_A_TSCONFIG_RX_FILTERS`, `ETHTOOL_A_TSCONFIG_HWTSTAMP_FLAGS` attribute가 구조체를 설정합니다. Nested `ETHTOOL_A_TSCONFIG_HWTSTAMP_PROVIDER`는 device source index와 timestamp type qualifier로 hardware source를 고릅니다.

Driver는 요청보다 더 permissive한 설정을 사용할 수 있고 hardware가 직접 지원하는 가장 generic mode를 구현하는 것이 좋습니다. 예를 들어 `HWTSTAMP_FILTER_PTP_V2_EVENT`를 지원하면 더 좁은 `HWTSTAMP_FILTER_PTP_V2_L2_SYNC` 요청을 generic event mode로 확장하는 편이 유용합니다.

Hardware timestamp driver는 실제 설정으로 구조체를 갱신해야 합니다. 요청 packet을 timestamp할 수 없으면 아무것도 바꾸지 않고 `ERANGE`를 반환합니다. `EINVAL`은 `SIOCSHWTSTAMP` 자체를 지원하지 않는다는 뜻입니다.

Admin 권한 process만 설정을 바꿀 수 있으며 여러 process 충돌 방지와 설정 reset은 userspace 책임입니다. 어떤 process든 `ETHTOOL_MSG_TSCONFIG_GET`으로 실제 설정을 읽을 수 있습니다.

Legacy 방식은 `ifreq.ifr_data`가 `hwtstamp_config`를 가리키는 `ioctl(SIOCSHWTSTAMP)`입니다. `tx_type`과 `rx_filter`는 hint이며 fine-grained RX filter를 지원하지 않으면 더 많은 packet을 timestamp할 수 있습니다. `SIOCGHWTSTAMP` read 방식은 모든 driver에 구현되지는 않았습니다.

3. Hardware Timestamping configuration: ETHTOOL_MSG_TSCONFIG_SET/GET
====================================================================

Hardware time stamping must also be initialized for each device driver
that is expected to do hardware time stamping. The parameter is defined in
include/uapi/linux/net_tstamp.h as::

        struct hwtstamp_config {
                int flags;        /* no flags defined right now, must be zero */
                int tx_type;        /* HWTSTAMP_TX_* */
                int rx_filter;        /* HWTSTAMP_FILTER_* */
        };

Desired behavior is passed into the kernel and to a specific device by
calling the tsconfig netlink socket ``ETHTOOL_MSG_TSCONFIG_SET``.
The ``ETHTOOL_A_TSCONFIG_TX_TYPES``, ``ETHTOOL_A_TSCONFIG_RX_FILTERS`` and
``ETHTOOL_A_TSCONFIG_HWTSTAMP_FLAGS`` netlink attributes are then used to set
the struct hwtstamp_config accordingly.

The ``ETHTOOL_A_TSCONFIG_HWTSTAMP_PROVIDER`` netlink nested attribute is used
to select the source of the hardware time stamping. It is composed of an index
for the device source and a qualifier for the type of time stamping.

Drivers are free to use a more permissive configuration than the requested
configuration. It is expected that drivers should only implement directly the
most generic mode that can be supported. For example if the hardware can
support HWTSTAMP_FILTER_PTP_V2_EVENT, then it should generally always upscale
HWTSTAMP_FILTER_PTP_V2_L2_SYNC, and so forth, as HWTSTAMP_FILTER_PTP_V2_EVENT
is more generic (and more useful to applications).

A driver which supports hardware time stamping shall update the struct
with the actual, possibly more permissive configuration. If the
requested packets cannot be time stamped, then nothing should be
changed and ERANGE shall be returned (in contrast to EINVAL, which
indicates that SIOCSHWTSTAMP is not supported at all).

Only a processes with admin rights may change the configuration. User
space is responsible to ensure that multiple processes don't interfere
with each other and that the settings are reset.

Any process can read the actual configuration by requesting tsconfig netlink
socket ``ETHTOOL_MSG_TSCONFIG_GET``.

The legacy configuration is the use of the ioctl(SIOCSHWTSTAMP) with a pointer
to a struct ifreq whose ifr_data points to a struct hwtstamp_config.
The tx_type and rx_filter are hints to the driver what it is expected to do.
If the requested fine-grained filtering for incoming packets is not
supported, the driver may time stamp more than just the requested types
of packets. ioctl(SIOCGHWTSTAMP) is used in the same way as the
ioctl(SIOCSHWTSTAMP). However, this has not been implemented in all drivers.

::

    /* possible values for hwtstamp_config->tx_type */
    enum {
            /*
            * no outgoing packet will need hardware time stamping;
            * should a packet arrive which asks for it, no hardware
            * time stamping will be done
            */
            HWTSTAMP_TX_OFF,

            /*
            * enables hardware time stamping for outgoing packets;
            * the sender of the packet decides which are to be
            * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
            * before sending the packet
            */
            HWTSTAMP_TX_ON,
    };

    /* possible values for hwtstamp_config->rx_filter */
    enum {
            /* time stamp no incoming packet at all */
            HWTSTAMP_FILTER_NONE,

            /* time stamp any incoming packet */
            HWTSTAMP_FILTER_ALL,

            /* return value: time stamp all packets requested plus some others */
            HWTSTAMP_FILTER_SOME,

            /* PTP v1, UDP, any kind of event packet */
            HWTSTAMP_FILTER_PTP_V1_L4_EVENT,

            /* for the complete list of values, please check
            * the include file include/uapi/linux/net_tstamp.h
            */
    };

Device driver 구현 책임

Hardware timestamp driver는 `ndo_hwtstamp_set` NDO 또는 legacy `SIOCSHWTSTAMP`를 지원하고 실제 값으로 `hwtstamp_config`를 갱신해야 합니다. `ndo_hwtstamp_get` 또는 legacy `SIOCGHWTSTAMP`도 지원해야 합니다.

RX packet timestamp는 skb에 저장합니다. `skb_hwtstamps()`로 shared `struct skb_shared_hwtstamps`를 얻어 arbitrary origin 이후 duration인 `ktime_t hwtstamp`를 설정합니다.

TX에서는 `hard_start_xmit()`에서 `skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP`를 검사합니다. 가능하고 요청됐다면 `SKBTX_IN_PROGRESS`를 설정해 driver가 hardware timestamping을 수행 중임을 선언하고, 필요하면 다음 단계를 위해 skb pointer를 보관해 아직 free하지 않습니다. Hardware timestamp를 지원하지 않는 driver는 이 flag를 설정하지 않습니다.

Driver는 network subsystem의 software timestamp 저장소인 `sk_buff::tstamp`를 절대 건드리면 안 됩니다.

Hardware에 skb를 넘기기 직전 가능한 가까운 곳에서 `skb_tx_timestamp()`를 호출합니다. 요청됐지만 `SKBTX_IN_PROGRESS`가 없어 hardware가 불가능하면 이 함수가 software timestamp를 제공합니다.

Packet 전송 또는 hardware timestamp 획득 뒤 original skb와 raw timestamp로 `skb_tstamp_tx()`를 호출합니다. 이 함수가 original skb를 clone하고 timestamp를 붙이므로 original은 이제 free해야 합니다. Hardware timestamp 획득 실패 뒤 software timestamp로 fallback하면 pipeline의 늦은 지점에서 찍혀 다른 software timestamp와 예상 밖 delta를 만들 수 있으므로 fallback하면 안 됩니다.

3.1 Hardware Timestamping Implementation: Device Drivers
--------------------------------------------------------

A driver which supports hardware time stamping must support the
ndo_hwtstamp_set NDO or the legacy SIOCSHWTSTAMP ioctl and update the
supplied struct hwtstamp_config with the actual values as described in
the section on SIOCSHWTSTAMP. It should also support ndo_hwtstamp_get or
the legacy SIOCGHWTSTAMP.

Time stamps for received packets must be stored in the skb. To get a pointer
to the shared time stamp structure of the skb call skb_hwtstamps(). Then
set the time stamps in the structure::

    struct skb_shared_hwtstamps {
            /* hardware time stamp transformed into duration
            * since arbitrary point in time
            */
            ktime_t        hwtstamp;
    };

Time stamps for outgoing packets are to be generated as follows:

- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
  is set no-zero. If yes, then the driver is expected to do hardware time
  stamping.
- If this is possible for the skb and requested, then declare
  that the driver is doing the time stamping by setting the flag
  SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with::

      skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;

  You might want to keep a pointer to the associated skb for the next step
  and not free the skb. A driver not supporting hardware time stamping doesn't
  do that. A driver must never touch sk_buff::tstamp! It is used to store
  software generated time stamps by the network subsystem.
- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
  as possible. skb_tx_timestamp() provides a software time stamp if requested
  and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
- As soon as the driver has sent the packet and/or obtained a
  hardware time stamp for it, it passes the time stamp back by
  calling skb_tstamp_tx() with the original skb, the raw
  hardware time stamp. skb_tstamp_tx() clones the original skb and
  adds the timestamps, therefore the original skb has to be freed now.
  If obtaining the hardware time stamp somehow fails, then the driver
  should not fall back to software time stamping. The rationale is that
  this would occur at a later time in the processing pipeline than other
  software time stamping and therefore could lead to unexpected deltas
  between time stamps.

Stacked PTP Hardware Clock 원칙

Packet data path에 PHC(PTP Hardware Clock)가 여러 개 있을 수 있습니다. Kernel에는 Ethernet frame timestamping에 사용할 PHC를 userspace가 직접 고르는 mechanism이 없습니다. 대신 wire에 가장 바깥쪽인 PHC가 항상 가장 바람직하다고 가정하고 driver들이 협력합니다. 현재 stacked PHC case는 DSA switch, Ethernet PHY, MII bus snooping device 세 가지입니다.

3.2 Special considerations for stacked PTP Hardware Clocks
----------------------------------------------------------

There are situations when there may be more than one PHC (PTP Hardware Clock)
in the data path of a packet. The kernel has no explicit mechanism to allow the
user to select which PHC to use for timestamping Ethernet frames. Instead, the
assumption is that the outermost PHC is always the most preferable, and that
kernel drivers collaborate towards achieving that goal. Currently there are 3
cases of stacked PHCs, detailed below:

DSA switch의 outermost PHC

DSA switch는 port 하나가 PTP를 모르는 host Ethernet interface에 연결되고 port multiplier와 optional forwarding acceleration 역할을 합니다. 각 switch port는 독립 virtual network interface로 보이지만 실제 I/O는 TX에서 host port로 redirect되고 RX frame을 intercept하는 방식으로 host interface를 통합니다.

Switch의 variable queue delay는 host port와 PTP partner 사이 path jitter를 만들어 synchronization을 해칩니다. 일부 DSA switch는 자체 clock과 MAC timestamping으로 wire와 PHY propagation latency만 측정합니다. Linux는 이를 일반 network interface ABI로 제공하며 virtual I/O port여도 자체 PHC를 가집니다. 보통 switch port들이 PHC를 공유하지만 필수는 아닙니다.

DSA switch를 붙인 host port driver에는 특별한 처리가 필요 없습니다. Host port도 PTP를 지원하면 DSA가 host의 `.ndo_eth_ioctl` 호출을 intercept해 hardware timestamp enable을 막습니다. `SO_TIMESTAMPING` API는 packet 하나에 hardware timestamp 여러 개를 전달할 수 없으므로 DSA port만 생성해야 합니다.

`.port_txtstamp()`는 userspace가 hardware TX timestamp를 요청한 packet 전송 전에 호출됩니다. Two-step timestamp는 MAC 전송 뒤에 값이 나오므로 original packet과 값을 연계해 error queue로 다시 넣어야 합니다. Driver는 `skb_clone_sk()`로 clone해 `skb->cb`와 TX skb queue에 보관할 수 있습니다.

Switch의 timestamp register/FIFO가 PTP sequence ID, message type, domain과 timestamp pair를 제공할 수 있습니다. Driver는 `ptp_classify_raw()`로 transport를 분류하고 `ptp_parse_header()`로 header를 읽어 대기 packet과 값을 연계합니다. Timestamp IRQ를 쓰거나 host interface로 `dev_queue_xmit()`한 뒤 poll할 수 있습니다.

One-step TX는 MAC이 packet에 timestamp를 넣고 PTP follow-up message가 필요 없으므로 clone할 필요가 없고 userspace도 annotated packet의 error queue 반환을 기대하지 않습니다.

`.port_rxtstamp()`에서 DSA는 BPF classifier로 PTP event message만 식별합니다. Driver는 original skb에 즉시 timestamp를 붙이거나 나중으로 미룹니다. Timestamp는 DSA header metadata 같은 in-band 방식이나 RX FIFO 같은 out-of-band 방식일 수 있습니다. Sleepable context가 필요해 defer했다면 DSA driver가 새 timestamp를 붙인 skb에 `netif_rx()`를 호출해야 합니다.

3.2.1 DSA (Distributed Switch Architecture) switches
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These are Ethernet switches which have one of their ports connected to an
(otherwise completely unaware) host Ethernet interface, and perform the role of
a port multiplier with optional forwarding acceleration features.  Each DSA
switch port is visible to the user as a standalone (virtual) network interface,
and its network I/O is performed, under the hood, indirectly through the host
interface (redirecting to the host port on TX, and intercepting frames on RX).

When a DSA switch is attached to a host port, PTP synchronization has to
suffer, since the switch's variable queuing delay introduces a path delay
jitter between the host port and its PTP partner. For this reason, some DSA
switches include a timestamping clock of their own, and have the ability to
perform network timestamping on their own MAC, such that path delays only
measure wire and PHY propagation latencies. Timestamping DSA switches are
supported in Linux and expose the same ABI as any other network interface (save
for the fact that the DSA interfaces are in fact virtual in terms of network
I/O, they do have their own PHC).  It is typical, but not mandatory, for all
interfaces of a DSA switch to share the same PHC.

By design, PTP timestamping with a DSA switch does not need any special
handling in the driver for the host port it is attached to.  However, when the
host port also supports PTP timestamping, DSA will take care of intercepting
the ``.ndo_eth_ioctl`` calls towards the host port, and block attempts to enable
hardware timestamping on it. This is because the SO_TIMESTAMPING API does not
allow the delivery of multiple hardware timestamps for the same packet, so
anybody else except for the DSA switch port must be prevented from doing so.

In the generic layer, DSA provides the following infrastructure for PTP
timestamping:

- ``.port_txtstamp()``: a hook called prior to the transmission of
  packets with a hardware TX timestamping request from user space.
  This is required for two-step timestamping, since the hardware
  timestamp becomes available after the actual MAC transmission, so the
  driver must be prepared to correlate the timestamp with the original
  packet so that it can re-enqueue the packet back into the socket's
  error queue. To save the packet for when the timestamp becomes
  available, the driver can call ``skb_clone_sk`` , save the clone pointer
  in skb->cb and enqueue a tx skb queue. Typically, a switch will have a
  PTP TX timestamp register (or sometimes a FIFO) where the timestamp
  becomes available. In case of a FIFO, the hardware might store
  key-value pairs of PTP sequence ID/message type/domain number and the
  actual timestamp. To perform the correlation correctly between the
  packets in a queue waiting for timestamping and the actual timestamps,
  drivers can use a BPF classifier (``ptp_classify_raw``) to identify
  the PTP transport type, and ``ptp_parse_header`` to interpret the PTP
  header fields. There may be an IRQ that is raised upon this
  timestamp's availability, or the driver might have to poll after
  invoking ``dev_queue_xmit()`` towards the host interface.
  One-step TX timestamping do not require packet cloning, since there is
  no follow-up message required by the PTP protocol (because the
  TX timestamp is embedded into the packet by the MAC), and therefore
  user space does not expect the packet annotated with the TX timestamp
  to be re-enqueued into its socket's error queue.

- ``.port_rxtstamp()``: On RX, the BPF classifier is run by DSA to
  identify PTP event messages (any other packets, including PTP general
  messages, are not timestamped). The original (and only) timestampable
  skb is provided to the driver, for it to annotate it with a timestamp,
  if that is immediately available, or defer to later. On reception,
  timestamps might either be available in-band (through metadata in the
  DSA header, or attached in other ways to the packet), or out-of-band
  (through another RX timestamping FIFO). Deferral on RX is typically
  necessary when retrieving the timestamp needs a sleepable context. In
  that case, it is the responsibility of the DSA driver to call
  ``netif_rx()`` on the freshly timestamped skb.

Ethernet PHY timestamping

Ethernet PHY는 보통 Layer 1 장치라 DSA switch처럼 network interface 표현이 없습니다. 그래도 wire에 가까운 timestamp가 더 안정적이고 정밀할 수 있어 PTP packet을 감지·timestamp할 수 있습니다.

PTP timestamping PHY driver는 `struct mii_timestamper`를 만들고 pointer를 `phydev->mii_ts`에 넣어 network stack이 발견하게 합니다.

PHY에는 network interface가 없으므로 timestamping과 ethtool ioctl은 MAC driver가 중계해야 합니다. 각 MAC driver는 `.ndo_eth_ioctl`에서 `phy_has_hwtstamp(netdev->phydev)`를 검사하고 true면 직접 처리하지 말고 `phy_mii_ioctl()`로 PHY에 넘깁니다.

Plain `netif_rx()` 계열 RX에서는 MAC이 `skb_defer_rx_timestamp(skb)`가 필요한지 검사하고 필요하면 `netif_rx()`를 호출하지 않습니다. `CONFIG_NETWORK_PHY_TIMESTAMPING`과 `skb->dev->phydev->mii_ts`가 있으면 `.rxtstamp()`가 defer 필요성을 판단하며, defer 시 timestamp 준비 뒤 stack에 packet을 올리는 것은 PHY driver 책임입니다.

`napi_gro_receive()`나 `netif_receive_skb()`는 stack이 자동으로 defer를 검사하므로 MAC driver 검사가 필요 없습니다.

TX hook `mii_ts->txtstamp()`는 `skb_clone_tx_timestamp()`가 호출합니다. MAC이 직접 호출할 수도 있지만 많은 MAC driver가 software timestamp를 위해 호출하는 `skb_tx_timestamp()`에서도 연계됩니다. 따라서 software timestamp를 지원하는 MAC은 보통 추가 동작이 필요 없습니다.

3.2.2 Ethernet PHYs
^^^^^^^^^^^^^^^^^^^

These are devices that typically fulfill a Layer 1 role in the network stack,
hence they do not have a representation in terms of a network interface as DSA
switches do. However, PHYs may be able to detect and timestamp PTP packets, for
performance reasons: timestamps taken as close as possible to the wire have the
potential to yield a more stable and precise synchronization.

A PHY driver that supports PTP timestamping must create a ``struct
mii_timestamper`` and add a pointer to it in ``phydev->mii_ts``. The presence
of this pointer will be checked by the networking stack.

Since PHYs do not have network interface representations, the timestamping and
ethtool ioctl operations for them need to be mediated by their respective MAC
driver.  Therefore, as opposed to DSA switches, modifications need to be done
to each individual MAC driver for PHY timestamping support. This entails:

- Checking, in ``.ndo_eth_ioctl``, whether ``phy_has_hwtstamp(netdev->phydev)``
  is true or not. If it is, then the MAC driver should not process this request
  but instead pass it on to the PHY using ``phy_mii_ioctl()``.

- On RX, special intervention may or may not be needed, depending on the
  function used to deliver skb's up the network stack. In the case of plain
  ``netif_rx()`` and similar, MAC drivers must check whether
  ``skb_defer_rx_timestamp(skb)`` is necessary or not - and if it is, don't
  call ``netif_rx()`` at all.  If ``CONFIG_NETWORK_PHY_TIMESTAMPING`` is
  enabled, and ``skb->dev->phydev->mii_ts`` exists, its ``.rxtstamp()`` hook
  will be called now, to determine, using logic very similar to DSA, whether
  deferral for RX timestamping is necessary.  Again like DSA, it becomes the
  responsibility of the PHY driver to send the packet up the stack when the
  timestamp is available.

  For other skb receive functions, such as ``napi_gro_receive`` and
  ``netif_receive_skb``, the stack automatically checks whether
  ``skb_defer_rx_timestamp()`` is necessary, so this check is not needed inside
  the driver.

- On TX, again, special intervention might or might not be needed.  The
  function that calls the ``mii_ts->txtstamp()`` hook is named
  ``skb_clone_tx_timestamp()``. This function can either be called directly
  (case in which explicit MAC driver support is indeed needed), but the
  function also piggybacks from the ``skb_tx_timestamp()`` call, which many MAC
  drivers already perform for software timestamping purposes. Therefore, if a
  MAC supports software timestamping, it does not need to do anything further
  at this stage.

MII bus snooping timestamp device

MII bus snooping device는 timestamping Ethernet PHY와 같은 역할을 하지만 discrete device라 timestamp 기능이 없는 어떤 PHY와도 함께 쓸 수 있습니다. Device Tree로 `struct phy_device`에 발견·연결되고 나머지는 같은 `mii_ts` infrastructure를 사용합니다. 자세한 binding path `Documentation/devicetree/bindings/ptp/timestamper.txt`를 보존합니다.

3.2.3 MII bus snooping devices
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These perform the same role as timestamping Ethernet PHYs, save for the fact
that they are discrete devices and can therefore be used in conjunction with
any PHY even if it doesn't support timestamping. In Linux, they are
discoverable and attachable to a ``struct phy_device`` through Device Tree, and
for the rest, they use the same mii_ts infrastructure as those. See
Documentation/devicetree/bindings/ptp/timestamper.txt for more details.

Stacked PHC에서 MAC duplicate 방지

Stacked PHC는 단독 MAC 환경에서는 드러나지 않던 driver bug를 노출할 수 있습니다. Plain MAC, DSA, PHY, MII snooper 등 TX timestamp logic은 모두 `skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS`를 설정할 수 있습니다. PHC stacking을 모르는 MAC이 자신이 아닌 entity가 설정한 flag를 자기 request로 오인하면 duplicate timestamp를 전달합니다.

일반적인 TX 단계는 `.ndo_eth_ioctl`로 PTP가 enable됐는지 `priv->hwtstamp_tx_enabled`를 보고 현재 skb에 `SKBTX_HW_TSTAMP` request가 있는지 검사해 `SKBTX_IN_PROGRESS`를 설정합니다. Stacked system에서는 MAC이 outermost PHC가 아니므로 이 조건은 발생하지 않아야 합니다.

문제는 TX confirmation에서 MAC이 timestamp 회수 필요성을 `SKBTX_IN_PROGRESS` 하나만 보고 판단하는 shortcut입니다. 이 flag는 data path의 다른 timestamp entity가 설정했을 수 있습니다.

올바른 해결은 confirmation에서 `SKBTX_IN_PROGRESS`뿐 아니라 `priv->hwtstamp_tx_enabled == true`도 함께 검사하는 것입니다. System은 outermost PHC 이외에는 PTP timestamp를 enable하지 않으므로 compound check가 duplicate TX timestamp의 userspace 전달을 막습니다.

3.2.4 Other caveats for MAC drivers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The use of stacked PHCs may uncover MAC driver bugs which were impossible to
trigger without them. One example has to do with this line of code, already
presented earlier::

      skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;

Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHY
driver or a MII bus snooping device driver, should set this flag.
But a MAC driver that is unaware of PHC stacking might get tripped up by
somebody other than itself setting this flag, and deliver a duplicate
timestamp.
For example, a typical driver design for TX timestamping might be to split the
transmission part into 2 portions:

1. "TX": checks whether PTP timestamping has been previously enabled through
   the ``.ndo_eth_ioctl`` ("``priv->hwtstamp_tx_enabled == true``") and the
   current skb requires a TX timestamp ("``skb_shinfo(skb)->tx_flags &
   SKBTX_HW_TSTAMP``"). If this is true, it sets the
   "``skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS``" flag. Note: as
   described above, in the case of a stacked PHC system, this condition should
   never trigger, as this MAC is certainly not the outermost PHC. But this is
   not where the typical issue is.  Transmission proceeds with this packet.

2. "TX confirmation": Transmission has finished. The driver checks whether it
   is necessary to collect any TX timestamp for it. Here is where the typical
   issues are: the MAC driver takes a shortcut and only checks whether
   "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``" was set. With a stacked
   PHC system, this is incorrect because this MAC driver is not the only entity
   in the TX data path who could have enabled SKBTX_IN_PROGRESS in the first
   place.

The correct solution for this problem is for MAC drivers to have a compound
check in their "TX confirmation" portion, not only for
"``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``", but also for
"``priv->hwtstamp_tx_enabled == true``". Because the rest of the system ensures
that PTP timestamping is not enabled for anything other than the outermost PHC,
this enhanced check will avoid delivering a duplicated TX timestamp to user
space.