요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
============================
Kernel NFS Server Statistics
============================
:Authors: Greg Banks <[email protected]> - 26 Mar 2009
This document describes the format and semantics of the statistics
which the kernel NFS server makes available to userspace. These
statistics are available in several text form pseudo files, each of
which is described separately below.
In most cases you don't need to know these formats, as the nfsstat(8)
program from the nfs-utils distribution provides a helpful command-line
interface for extracting and printing them.
All the files described here are formatted as a sequence of text lines,
separated by newline '\n' characters. Lines beginning with a hash
'#' character are comments intended for humans and should be ignored
by parsing routines. All other lines contain a sequence of fields
separated by whitespace.
/proc/fs/nfsd/pool_stats
========================
This file is available in kernels from 2.6.30 onwards, if the
/proc/fs/nfsd filesystem is mounted (it almost always should be).
The first line is a comment which describes the fields present in
all the other lines. The other lines present the following data as
a sequence of unsigned decimal numeric fields. One line is shown
for each NFS thread pool.
All counters are 64 bits wide and wrap naturally. There is no way
to zero these counters, instead applications should do their own
rate conversion.
pool
The id number of the NFS thread pool to which this line applies.
This number does not change.
Thread pool ids are a contiguous set of small integers starting
at zero. The maximum value depends on the thread pool mode, but
currently cannot be larger than the number of CPUs in the system.
Note that in the default case there will be a single thread pool
which contains all the nfsd threads and all the CPUs in the system,
and thus this file will have a single line with a pool id of "0".
packets-arrived
Counts how many NFS packets have arrived. More precisely, this
is the number of times that the network stack has notified the
sunrpc server layer that new data may be available on a transport
(e.g. an NFS or UDP socket or an NFS/RDMA endpoint).
Depending on the NFS workload patterns and various network stack
effects (such as Large Receive Offload) which can combine packets
on the wire, this may be either more or less than the number
of NFS calls received (which statistic is available elsewhere).
However this is a more accurate and less workload-dependent measure
of how much CPU load is being placed on the sunrpc server layer
due to NFS network traffic.
sockets-enqueued
Counts how many times an NFS transport is enqueued to wait for
an nfsd thread to service it, i.e. no nfsd thread was considered
available.
The circumstance this statistic tracks indicates that there was NFS
network-facing work to be done but it couldn't be done immediately,
thus introducing a small delay in servicing NFS calls. The ideal
rate of change for this counter is zero; significantly non-zero
values may indicate a performance limitation.
This can happen because there are too few nfsd threads in the thread
pool for the NFS workload (the workload is thread-limited), in which
case configuring more nfsd threads will probably improve the
performance of the NFS workload.
threads-woken
Counts how many times an idle nfsd thread is woken to try to
receive some data from an NFS transport.
This statistic tracks the circumstance where incoming
network-facing NFS work is being handled quickly, which is a good
thing. The ideal rate of change for this counter will be close
to but less than the rate of change of the packets-arrived counter.
threads-timedout
Counts how many times an nfsd thread triggered an idle timeout,
i.e. was not woken to handle any incoming network packets for
some time.
This statistic counts a circumstance where there are more nfsd
threads configured than can be used by the NFS workload. This is
a clue that the number of nfsd threads can be reduced without
affecting performance. Unfortunately, it's only a clue and not
a strong indication, for a couple of reasons:
- Currently the rate at which the counter is incremented is quite
slow; the idle timeout is 60 minutes. Unless the NFS workload
remains constant for hours at a time, this counter is unlikely
to be providing information that is still useful.
- It is usually a wise policy to provide some slack,
i.e. configure a few more nfsds than are currently needed,
to allow for future spikes in load.
Note that incoming packets on NFS transports will be dealt with in
one of three ways. An nfsd thread can be woken (threads-woken counts
this case), or the transport can be enqueued for later attention
(sockets-enqueued counts this case), or the packet can be temporarily
deferred because the transport is currently being used by an nfsd
thread. This last case is not very interesting and is not explicitly
counted, but can be inferred from the other counters thus::
packets-deferred = packets-arrived - ( sockets-enqueued + threads-woken )
More
====
Descriptions of the other statistics file should go here.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
Kernel NFS server 통계 파일의 공통 형식
1-21Greg Banks가 2009년 3월 26일 작성한 이 문서는 kernel NFS server가 user space에 제공하는 통계의 형식과 의미를 설명합니다. 통계는 여러 text pseudo-file에 나뉘며 각 파일의 형식은 별도로 정의됩니다.
대부분의 사용자는 이 raw 형식을 직접 알 필요가 없습니다. `nfs-utils` 배포판의 `nfsstat(8)`가 통계를 추출하고 출력하는 command-line interface를 제공합니다.
여기서 설명하는 파일은 newline `\n`으로 구분한 text line의 sequence입니다. `#`으로 시작하는 line은 사람을 위한 comment이므로 parser가 무시해야 합니다. 그 밖의 line은 whitespace로 구분한 field sequence입니다.
Pseudo-file의 raw field와 일반 관리 도구의 관계입니다.
============================
Kernel NFS Server Statistics
============================
:Authors: Greg Banks <[email protected]> - 26 Mar 2009
This document describes the format and semantics of the statistics
which the kernel NFS server makes available to userspace. These
statistics are available in several text form pseudo files, each of
which is described separately below.
In most cases you don't need to know these formats, as the nfsstat(8)
program from the nfs-utils distribution provides a helpful command-line
interface for extracting and printing them.
All the files described here are formatted as a sequence of text lines,
separated by newline '\n' characters. Lines beginning with a hash
'#' character are comments intended for humans and should be ignored
by parsing routines. All other lines contain a sequence of fields
separated by whitespace.
/proc/fs/nfsd/pool_stats와 pool field
22-46`/proc/fs/nfsd/pool_stats`는 `/proc/fs/nfsd` filesystem이 mount된 Linux 2.6.30 이후 kernel에서 사용할 수 있습니다. 첫 line은 뒤따르는 모든 line의 field를 설명하는 comment이고, 각 NFS thread pool마다 unsigned decimal numeric field로 구성된 data line 하나가 나옵니다.
모든 counter는 64-bit이며 overflow 시 자연스럽게 wrap합니다. Counter를 0으로 만드는 interface는 없으므로 application이 두 시점의 차이를 시간으로 나누어 자체적으로 rate를 계산해야 합니다.
`pool` field는 그 line이 나타내는 NFS thread pool의 고정 ID입니다. ID는 0부터 시작하는 연속된 작은 정수이고 최대값은 thread-pool mode에 따라 달라지지만 현재는 system CPU 수보다 클 수 없습니다.
기본 구성에서는 system의 모든 nfsd thread와 CPU를 포함하는 thread pool 하나만 존재하므로 파일에는 `pool` ID가 `0`인 data line 하나가 나타납니다.
Header와 pool별 data line의 의미입니다.
/proc/fs/nfsd/pool_stats
========================
This file is available in kernels from 2.6.30 onwards, if the
/proc/fs/nfsd filesystem is mounted (it almost always should be).
The first line is a comment which describes the fields present in
all the other lines. The other lines present the following data as
a sequence of unsigned decimal numeric fields. One line is shown
for each NFS thread pool.
All counters are 64 bits wide and wrap naturally. There is no way
to zero these counters, instead applications should do their own
rate conversion.
pool
The id number of the NFS thread pool to which this line applies.
This number does not change.
Thread pool ids are a contiguous set of small integers starting
at zero. The maximum value depends on the thread pool mode, but
currently cannot be larger than the number of CPUs in the system.
Note that in the default case there will be a single thread pool
which contains all the nfsd threads and all the CPUs in the system,
and thus this file will have a single line with a pool id of "0".packets-arrived와 sockets-enqueued
47-77`packets-arrived`는 NFS packet이 도착한 횟수를 셉니다. 정확히는 network stack이 SUNRPC server layer에 transport에서 새 data를 읽을 수 있다고 알린 횟수입니다. Transport는 NFS 또는 UDP socket이나 NFS/RDMA endpoint일 수 있습니다.
NFS workload pattern과 Large Receive Offload처럼 wire packet을 합치는 network-stack 효과 때문에 이 값은 수신한 NFS call 수보다 많거나 적을 수 있습니다. 그럼에도 NFS network traffic이 SUNRPC server layer에 가하는 CPU load를 call 수보다 정확하고 workload 의존성이 낮게 나타냅니다.
`sockets-enqueued`는 service할 수 있는 nfsd thread가 없다고 판단되어 NFS transport가 queue에 들어간 횟수입니다. Network-facing NFS work가 있었지만 즉시 처리하지 못해 call service에 작은 delay가 생겼다는 뜻입니다.
이 counter의 이상적인 변화율은 0입니다. 유의미한 증가가 계속되면 performance limitation일 수 있으며, workload에 비해 pool의 nfsd thread가 너무 적은 thread-limited 상황이라면 nfsd thread 수를 늘려 성능을 개선할 가능성이 큽니다.
도착 부하와 즉시 처리 실패를 구분합니다.
packets-arrived
Counts how many NFS packets have arrived. More precisely, this
is the number of times that the network stack has notified the
sunrpc server layer that new data may be available on a transport
(e.g. an NFS or UDP socket or an NFS/RDMA endpoint).
Depending on the NFS workload patterns and various network stack
effects (such as Large Receive Offload) which can combine packets
on the wire, this may be either more or less than the number
of NFS calls received (which statistic is available elsewhere).
However this is a more accurate and less workload-dependent measure
of how much CPU load is being placed on the sunrpc server layer
due to NFS network traffic.
sockets-enqueued
Counts how many times an NFS transport is enqueued to wait for
an nfsd thread to service it, i.e. no nfsd thread was considered
available.
The circumstance this statistic tracks indicates that there was NFS
network-facing work to be done but it couldn't be done immediately,
thus introducing a small delay in servicing NFS calls. The ideal
rate of change for this counter is zero; significantly non-zero
values may indicate a performance limitation.
This can happen because there are too few nfsd threads in the thread
pool for the NFS workload (the workload is thread-limited), in which
case configuring more nfsd threads will probably improve the
performance of the NFS workload.
threads-woken과 threads-timedout
78-107`threads-woken`은 idle nfsd thread를 깨워 NFS transport에서 data를 받도록 시도한 횟수입니다. 들어오는 network-facing NFS work를 빠르게 처리한 경우이므로 좋은 신호이며, 이상적인 변화율은 `packets-arrived` 변화율에 가깝되 그보다 조금 작습니다.
`threads-timedout`은 nfsd thread가 일정 시간 incoming network packet 처리를 위해 깨어나지 못해 idle timeout을 발생시킨 횟수입니다. Workload가 사용할 수 있는 것보다 nfsd thread를 많이 구성했을 가능성을 보여 주므로 성능 영향 없이 thread 수를 줄일 수 있다는 단서입니다.
다만 강한 결론으로 사용하면 안 됩니다. 현재 idle timeout은 60분이라 counter 증가가 매우 느리고, workload가 여러 시간 일정하지 않으면 값이 이미 낡은 상황을 반영할 수 있습니다. 또한 향후 load spike를 흡수하도록 현재 필요량보다 nfsd를 조금 더 두는 것이 보통 현명한 운영 정책입니다.
빠른 dispatch와 장시간 미사용 thread를 구분합니다.
threads-woken
Counts how many times an idle nfsd thread is woken to try to
receive some data from an NFS transport.
This statistic tracks the circumstance where incoming
network-facing NFS work is being handled quickly, which is a good
thing. The ideal rate of change for this counter will be close
to but less than the rate of change of the packets-arrived counter.
threads-timedout
Counts how many times an nfsd thread triggered an idle timeout,
i.e. was not woken to handle any incoming network packets for
some time.
This statistic counts a circumstance where there are more nfsd
threads configured than can be used by the NFS workload. This is
a clue that the number of nfsd threads can be reduced without
affecting performance. Unfortunately, it's only a clue and not
a strong indication, for a couple of reasons:
- Currently the rate at which the counter is incremented is quite
slow; the idle timeout is 60 minutes. Unless the NFS workload
remains constant for hours at a time, this counter is unlikely
to be providing information that is still useful.
- It is usually a wise policy to provide some slack,
i.e. configure a few more nfsds than are currently needed,
to allow for future spikes in load.
Transport 처리 세 경로와 packets-deferred
108-122NFS transport에 들어온 packet은 세 방식 가운데 하나로 처리됩니다. Idle nfsd thread를 깨우면 `threads-woken`, 가용 thread가 없어 transport를 나중 처리를 위해 queue에 넣으면 `sockets-enqueued`가 증가합니다.
세 번째는 transport를 현재 다른 nfsd thread가 사용 중이어서 packet 처리를 잠시 defer하는 경우입니다. 이 사건은 별도 counter로 기록하지 않지만 다른 counter의 차이로 계산할 수 있습니다.
packets-deferred = packets-arrived - (sockets-enqueued + threads-woken)마지막 `More` 절은 다른 statistics file의 설명이 이 위치에 추가되어야 한다는 placeholder입니다. 따라서 현재 문서는 `/proc/fs/nfsd/pool_stats` 형식과 field에 집중합니다.
`packets-arrived`를 두 직접 counter와 한 추론 값으로 분해합니다.
Note that incoming packets on NFS transports will be dealt with in
one of three ways. An nfsd thread can be woken (threads-woken counts
this case), or the transport can be enqueued for later attention
(sockets-enqueued counts this case), or the packet can be temporarily
deferred because the transport is currently being used by an nfsd
thread. This last case is not very interesting and is not explicitly
counted, but can be inferred from the other counters thus::
packets-deferred = packets-arrived - ( sockets-enqueued + threads-woken )
More
====
Descriptions of the other statistics file should go here.
요약·해설
knfsd-stats.rst:1-122`/proc/fs/nfsd/pool_stats`는 thread pool별 64-bit 누적 counter를 제공합니다. `packets-arrived`는 network-facing load, `sockets-enqueued`는 즉시 처리할 thread 부족, `threads-woken`은 빠른 dispatch, `threads-timedout`은 장시간 사용되지 않은 여유 thread를 나타냅니다.
Counter는 reset할 수 없고 wrap할 수 있으므로 monitoring application이 자체적으로 시간당 변화율을 계산해야 합니다. `packets-deferred`는 도착 수에서 enqueue와 wake 수를 뺀 값으로 추론합니다.
주요 counter를 capacity 판단과 연결합니다.