요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
Started Nov 1999 by Kanoj Sarcar <[email protected]>
=============
What is NUMA?
=============
This question can be answered from a couple of perspectives: the
hardware view and the Linux software view.
From the hardware perspective, a NUMA system is a computer platform that
comprises multiple components or assemblies each of which may contain 0
or more CPUs, local memory, and/or IO buses. For brevity and to
disambiguate the hardware view of these physical components/assemblies
from the software abstraction thereof, we'll call the components/assemblies
'cells' in this document.
Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset
of the system--although some components necessary for a stand-alone SMP system
may not be populated on any given cell. The cells of the NUMA system are
connected together with some sort of system interconnect--e.g., a crossbar or
point-to-point link are common types of NUMA system interconnects. Both of
these types of interconnects can be aggregated to create NUMA platforms with
cells at multiple distances from other cells.
For Linux, the NUMA platforms of interest are primarily what is known as Cache
Coherent NUMA or ccNUMA systems. With ccNUMA systems, all memory is visible
to and accessible from any CPU attached to any cell and cache coherency
is handled in hardware by the processor caches and/or the system interconnect.
Memory access time and effective memory bandwidth varies depending on how far
away the cell containing the CPU or IO bus making the memory access is from the
cell containing the target memory. For example, access to memory by CPUs
attached to the same cell will experience faster access times and higher
bandwidths than accesses to memory on other, remote cells. NUMA platforms
can have cells at multiple remote distances from any given cell.
Platform vendors don't build NUMA systems just to make software developers'
lives interesting. Rather, this architecture is a means to provide scalable
memory bandwidth. However, to achieve scalable memory bandwidth, system and
application software must arrange for a large majority of the memory references
[cache misses] to be to "local" memory--memory on the same cell, if any--or
to the closest cell with memory.
This leads to the Linux software view of a NUMA system:
Linux divides the system's hardware resources into multiple software
abstractions called "nodes". Linux maps the nodes onto the physical cells
of the hardware platform, abstracting away some of the details for some
architectures. As with physical cells, software nodes may contain 0 or more
CPUs, memory and/or IO buses. And, again, memory accesses to memory on
"closer" nodes--nodes that map to closer cells--will generally experience
faster access times and higher effective bandwidth than accesses to more
remote cells.
For some architectures, such as x86, Linux will "hide" any node representing a
physical cell that has no memory attached, and reassign any CPUs attached to
that cell to a node representing a cell that does have memory. Thus, on
these architectures, one cannot assume that all CPUs that Linux associates with
a given node will see the same local memory access times and bandwidth.
In addition, for some architectures, again x86 is an example, Linux supports
the emulation of additional nodes. For NUMA emulation, linux will carve up
the existing nodes--or the system memory for non-NUMA platforms--into multiple
nodes. Each emulated node will manage a fraction of the underlying cells'
physical memory. NUMA emulation is useful for testing NUMA kernel and
application features on non-NUMA platforms, and as a sort of memory resource
management mechanism when used together with cpusets.
[see Documentation/admin-guide/cgroup-v1/cpusets.rst]
For each node with memory, Linux constructs an independent memory management
subsystem, complete with its own free page lists, in-use page lists, usage
statistics and locks to mediate access. In addition, Linux constructs for
each memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE],
an ordered "zonelist". A zonelist specifies the zones/nodes to visit when a
selected zone/node cannot satisfy the allocation request. This situation,
when a zone has no available memory to satisfy a request, is called
"overflow" or "fallback".
Because some nodes contain multiple zones containing different types of
memory, Linux must decide whether to order the zonelists such that allocations
fall back to the same zone type on a different node, or to a different zone
type on the same node. This is an important consideration because some zones,
such as DMA or DMA32, represent relatively scarce resources. Linux chooses
a default Node ordered zonelist. This means it tries to fallback to other zones
from the same node before using remote nodes which are ordered by NUMA distance.
By default, Linux will attempt to satisfy memory allocation requests from the
node to which the CPU that executes the request is assigned. Specifically,
Linux will attempt to allocate from the first node in the appropriate zonelist
for the node where the request originates. This is called "local allocation."
If the "local" node cannot satisfy the request, the kernel will examine other
nodes' zones in the selected zonelist looking for the first zone in the list
that can satisfy the request.
Local allocation will tend to keep subsequent access to the allocated memory
"local" to the underlying physical resources and off the system interconnect--
as long as the task on whose behalf the kernel allocated some memory does not
later migrate away from that memory. The Linux scheduler is aware of the
NUMA topology of the platform--embodied in the "scheduling domains" data
structures [see Documentation/scheduler/sched-domains.rst]--and the scheduler
attempts to minimize task migration to distant scheduling domains. However,
the scheduler does not take a task's NUMA footprint into account directly.
Thus, under sufficient imbalance, tasks can migrate between nodes, remote
from their initial node and kernel data structures.
System administrators and application designers can restrict a task's migration
to improve NUMA locality using various CPU affinity command line interfaces,
such as taskset(1) and numactl(1), and program interfaces such as
sched_setaffinity(2). Further, one can modify the kernel's default local
allocation behavior using Linux NUMA memory policy. [see
Documentation/admin-guide/mm/numa_memory_policy.rst].
System administrators can restrict the CPUs and nodes' memories that a non-
privileged user can specify in the scheduling or NUMA commands and functions
using control groups and CPUsets. [see Documentation/admin-guide/cgroup-v1/cpusets.rst]
On architectures that do not hide memoryless nodes, Linux will include only
zones [nodes] with memory in the zonelists. This means that for a memoryless
node the "local memory node"--the node of the first zone in CPU's node's
zonelist--will not be the node itself. Rather, it will be the node that the
kernel selected as the nearest node with memory when it built the zonelists.
So, default, local allocations will succeed with the kernel supplying the
closest available memory. This is a consequence of the same mechanism that
allows such allocations to fallback to other nearby nodes when a node that
does contain memory overflows.
Some kernel allocations do not want or cannot tolerate this allocation fallback
behavior. Rather they want to be sure they get memory from the specified node
or get notified that the node has no free memory. This is usually the case when
a subsystem allocates per CPU memory resources, for example.
A typical model for making such an allocation is to obtain the node id of the
node to which the "current CPU" is attached using one of the kernel's
numa_node_id() or CPU_to_node() functions and then request memory from only
the node id returned. When such an allocation fails, the requesting subsystem
may revert to its own fallback path. The slab kernel memory allocator is an
example of this. Or, the subsystem may choose to disable or not to enable
itself on allocation failure. The kernel profiling subsystem is an example of
this.
If the architecture supports--does not hide--memoryless nodes, then CPUs
attached to memoryless nodes would always incur the fallback path overhead
or some subsystems would fail to initialize if they attempted to allocated
memory exclusively from a node without memory. To support such
architectures transparently, kernel subsystems can use the numa_mem_id()
or cpu_to_mem() function to locate the "local memory node" for the calling or
specified CPU. Again, this is the same node from which default, local page
allocations will be attempted.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
NUMA의 hardware 관점
1-43Kanoj Sarcar가 1999년 11월에 작성을 시작했습니다.
NUMA가 무엇인지는 hardware 관점과 Linux software 관점에서 답할 수 있습니다.
Hardware 관점에서 NUMA system은 여러 component 또는 assembly로 구성된 computer platform입니다. 각 component에는 CPU 0개 이상, local memory, I/O bus 중 일부가 들어 있을 수 있습니다. 이 문서에서는 physical component 또는 assembly라는 hardware 관점과 software abstraction을 구분하기 위해 이를 간단히 `cell`이라고 부릅니다.
각 cell은 system의 SMP, 즉 symmetric multiprocessor 부분 집합으로 볼 수 있습니다. 다만 독립 SMP system에 필요한 component 일부가 특정 cell에는 없을 수 있습니다. NUMA system의 cell은 crossbar나 point-to-point link 같은 system interconnect로 연결됩니다. 두 interconnect 모두 aggregation할 수 있어 다른 cell까지의 거리가 여러 단계인 NUMA platform을 만들 수 있습니다.
Linux가 주로 다루는 NUMA platform은 Cache Coherent NUMA, 즉 ccNUMA system입니다. ccNUMA에서는 어떤 cell의 CPU에서도 모든 memory를 볼 수 있고 접근할 수 있으며 processor cache 또는 system interconnect가 hardware에서 cache coherency를 처리합니다.
Memory access time과 유효 memory bandwidth는 접근을 수행하는 CPU 또는 I/O bus가 있는 cell과 target memory가 있는 cell 사이의 거리에 따라 달라집니다. 같은 cell에 연결된 CPU가 memory에 접근하면 remote cell memory에 접근할 때보다 latency가 짧고 bandwidth가 높습니다. 하나의 cell에서 여러 remote distance가 존재할 수 있습니다.
NUMA architecture의 목적은 software developer를 곤란하게 하는 것이 아니라 확장 가능한 memory bandwidth를 제공하는 것입니다. 이를 실현하려면 system과 application software가 memory reference, 즉 cache miss의 대부분을 같은 cell의 local memory 또는 memory가 있는 가장 가까운 cell로 향하게 해야 합니다.
Started Nov 1999 by Kanoj Sarcar <[email protected]>
=============
What is NUMA?
=============
This question can be answered from a couple of perspectives: the
hardware view and the Linux software view.
From the hardware perspective, a NUMA system is a computer platform that
comprises multiple components or assemblies each of which may contain 0
or more CPUs, local memory, and/or IO buses. For brevity and to
disambiguate the hardware view of these physical components/assemblies
from the software abstraction thereof, we'll call the components/assemblies
'cells' in this document.
Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset
of the system--although some components necessary for a stand-alone SMP system
may not be populated on any given cell. The cells of the NUMA system are
connected together with some sort of system interconnect--e.g., a crossbar or
point-to-point link are common types of NUMA system interconnects. Both of
these types of interconnects can be aggregated to create NUMA platforms with
cells at multiple distances from other cells.
For Linux, the NUMA platforms of interest are primarily what is known as Cache
Coherent NUMA or ccNUMA systems. With ccNUMA systems, all memory is visible
to and accessible from any CPU attached to any cell and cache coherency
is handled in hardware by the processor caches and/or the system interconnect.
Memory access time and effective memory bandwidth varies depending on how far
away the cell containing the CPU or IO bus making the memory access is from the
cell containing the target memory. For example, access to memory by CPUs
attached to the same cell will experience faster access times and higher
bandwidths than accesses to memory on other, remote cells. NUMA platforms
can have cells at multiple remote distances from any given cell.
Platform vendors don't build NUMA systems just to make software developers'
lives interesting. Rather, this architecture is a means to provide scalable
memory bandwidth. However, to achieve scalable memory bandwidth, system and
application software must arrange for a large majority of the memory references
[cache misses] to be to "local" memory--memory on the same cell, if any--or
to the closest cell with memory.
Linux node abstraction과 NUMA emulation
44-69Linux는 system의 hardware resource를 `node`라는 여러 software abstraction으로 나눕니다. Node를 hardware platform의 physical cell에 mapping하면서 architecture에 따라 세부 차이를 숨깁니다. Physical cell처럼 software node에도 CPU, memory, I/O bus가 0개 이상 있을 수 있습니다. 가까운 cell에 mapping된 가까운 node의 memory 접근은 remote cell보다 대체로 빠르고 유효 bandwidth가 높습니다.
x86 같은 일부 architecture에서 Linux는 memory가 없는 physical cell의 node를 숨기고 그 cell의 CPU를 memory가 있는 cell의 node에 다시 배정합니다. 따라서 이런 architecture에서는 Linux가 한 node에 연결한 모든 CPU가 같은 local-memory latency와 bandwidth를 가진다고 가정할 수 없습니다.
또한 x86 같은 일부 architecture는 추가 node emulation을 지원합니다. NUMA emulation은 기존 node 또는 non-NUMA platform의 system memory를 여러 node로 나눕니다. 각 emulated node는 기반 cell physical memory의 일부를 관리합니다. Non-NUMA platform에서 NUMA kernel과 application 기능을 검사할 때 유용하고, cpuset과 함께 사용하면 memory-resource management 수단이 됩니다. `Documentation/admin-guide/cgroup-v1/cpusets.rst`를 참조하십시오.
This leads to the Linux software view of a NUMA system:
Linux divides the system's hardware resources into multiple software
abstractions called "nodes". Linux maps the nodes onto the physical cells
of the hardware platform, abstracting away some of the details for some
architectures. As with physical cells, software nodes may contain 0 or more
CPUs, memory and/or IO buses. And, again, memory accesses to memory on
"closer" nodes--nodes that map to closer cells--will generally experience
faster access times and higher effective bandwidth than accesses to more
remote cells.
For some architectures, such as x86, Linux will "hide" any node representing a
physical cell that has no memory attached, and reassign any CPUs attached to
that cell to a node representing a cell that does have memory. Thus, on
these architectures, one cannot assume that all CPUs that Linux associates with
a given node will see the same local memory access times and bandwidth.
In addition, for some architectures, again x86 is an example, Linux supports
the emulation of additional nodes. For NUMA emulation, linux will carve up
the existing nodes--or the system memory for non-NUMA platforms--into multiple
nodes. Each emulated node will manage a fraction of the underlying cells'
physical memory. NUMA emulation is useful for testing NUMA kernel and
application features on non-NUMA platforms, and as a sort of memory resource
management mechanism when used together with cpusets.
[see Documentation/admin-guide/cgroup-v1/cpusets.rst]
Zonelist와 local allocation
70-105Linux는 memory가 있는 node마다 독립 memory-management subsystem을 만들며 각 subsystem은 자체 free-page list, in-use page list, usage statistic과 접근 조정용 lock을 가집니다. 또한 각 memory zone, 즉 DMA·DMA32·NORMAL·HIGH_MEMORY·MOVABLE 중 하나 이상에 순서가 있는 `zonelist`를 만듭니다. 선택한 zone 또는 node가 allocation request를 충족하지 못할 때 방문할 zone과 node를 zonelist가 지정합니다. Zone에 request를 충족할 memory가 없는 상태를 overflow 또는 fallback이라고 합니다.
일부 node에는 서로 다른 memory type의 zone이 여러 개 있으므로 allocation이 다른 node의 같은 zone type으로 fallback할지, 같은 node의 다른 zone type으로 fallback할지 정해야 합니다. DMA와 DMA32 같은 일부 zone은 비교적 희소한 resource라 이 선택이 중요합니다. Linux의 기본값은 node-ordered zonelist입니다. 먼저 같은 node의 다른 zone으로 fallback한 뒤 NUMA distance 순으로 remote node를 사용합니다.
기본적으로 Linux는 request를 실행하는 CPU가 배정된 node에서 memory-allocation request를 충족하려 합니다. 구체적으로 request가 시작된 node에 맞는 zonelist의 첫 node에서 할당하려고 합니다. 이를 local allocation이라고 합니다. Local node가 request를 충족하지 못하면 선택한 zonelist에 있는 다른 node의 zone을 차례로 검사해 처음으로 충족 가능한 zone을 사용합니다.
Kernel이 대신 memory를 할당한 task가 나중에 그 memory에서 멀어지지 않는 한 local allocation은 후속 접근을 기반 physical resource에 가깝게 유지하고 system interconnect 사용을 줄입니다. Linux scheduler는 `scheduling domains` 자료 구조에 표현된 platform NUMA topology를 알고 있으며, `Documentation/scheduler/sched-domains.rst`에서 설명하듯 task가 먼 scheduling domain으로 이동하는 것을 줄이려 합니다. 그러나 task의 NUMA footprint를 직접 고려하지는 않습니다. 따라서 imbalance가 충분히 크면 task가 초기 node와 kernel 자료 구조에서 먼 node로 migration될 수 있습니다.
For each node with memory, Linux constructs an independent memory management
subsystem, complete with its own free page lists, in-use page lists, usage
statistics and locks to mediate access. In addition, Linux constructs for
each memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE],
an ordered "zonelist". A zonelist specifies the zones/nodes to visit when a
selected zone/node cannot satisfy the allocation request. This situation,
when a zone has no available memory to satisfy a request, is called
"overflow" or "fallback".
Because some nodes contain multiple zones containing different types of
memory, Linux must decide whether to order the zonelists such that allocations
fall back to the same zone type on a different node, or to a different zone
type on the same node. This is an important consideration because some zones,
such as DMA or DMA32, represent relatively scarce resources. Linux chooses
a default Node ordered zonelist. This means it tries to fallback to other zones
from the same node before using remote nodes which are ordered by NUMA distance.
By default, Linux will attempt to satisfy memory allocation requests from the
node to which the CPU that executes the request is assigned. Specifically,
Linux will attempt to allocate from the first node in the appropriate zonelist
for the node where the request originates. This is called "local allocation."
If the "local" node cannot satisfy the request, the kernel will examine other
nodes' zones in the selected zonelist looking for the first zone in the list
that can satisfy the request.
Local allocation will tend to keep subsequent access to the allocated memory
"local" to the underlying physical resources and off the system interconnect--
as long as the task on whose behalf the kernel allocated some memory does not
later migrate away from that memory. The Linux scheduler is aware of the
NUMA topology of the platform--embodied in the "scheduling domains" data
structures [see Documentation/scheduler/sched-domains.rst]--and the scheduler
attempts to minimize task migration to distant scheduling domains. However,
the scheduler does not take a task's NUMA footprint into account directly.
Thus, under sufficient imbalance, tasks can migrate between nodes, remote
from their initial node and kernel data structures.
Affinity, NUMA policy와 memoryless node
106-126System administrator와 application designer는 `taskset(1)`, `numactl(1)` 같은 CPU-affinity command-line interface와 `sched_setaffinity(2)` 같은 program interface로 task migration을 제한해 NUMA locality를 개선할 수 있습니다. Linux NUMA memory policy로 kernel의 기본 local-allocation 동작도 바꿀 수 있습니다. `Documentation/admin-guide/mm/numa_memory_policy.rst`를 참조하십시오.
Control group과 cpuset으로 non-privileged user가 scheduling 또는 NUMA command와 function에서 지정할 수 있는 CPU 및 node memory를 제한할 수 있습니다. `Documentation/admin-guide/cgroup-v1/cpusets.rst`를 참조하십시오.
Memoryless node를 숨기지 않는 architecture에서 Linux는 memory가 있는 zone과 node만 zonelist에 넣습니다. 따라서 memoryless node의 `local memory node`, 즉 CPU node의 zonelist 첫 zone이 속한 node는 자기 자신이 아닙니다. Zonelist를 만들 때 kernel이 선택한 memory가 있는 가장 가까운 node입니다. 이 덕분에 기본 local allocation은 가장 가까운 available memory를 받아 성공합니다. Memory가 있는 node가 overflow할 때 가까운 다른 node로 fallback하는 것과 같은 mechanism입니다.
System administrators and application designers can restrict a task's migration
to improve NUMA locality using various CPU affinity command line interfaces,
such as taskset(1) and numactl(1), and program interfaces such as
sched_setaffinity(2). Further, one can modify the kernel's default local
allocation behavior using Linux NUMA memory policy. [see
Documentation/admin-guide/mm/numa_memory_policy.rst].
System administrators can restrict the CPUs and nodes' memories that a non-
privileged user can specify in the scheduling or NUMA commands and functions
using control groups and CPUsets. [see Documentation/admin-guide/cgroup-v1/cpusets.rst]
On architectures that do not hide memoryless nodes, Linux will include only
zones [nodes] with memory in the zonelists. This means that for a memoryless
node the "local memory node"--the node of the first zone in CPU's node's
zonelist--will not be the node itself. Rather, it will be the node that the
kernel selected as the nearest node with memory when it built the zonelists.
So, default, local allocations will succeed with the kernel supplying the
closest available memory. This is a consequence of the same mechanism that
allows such allocations to fallback to other nearby nodes when a node that
does contain memory overflows.
지정 node 전용 allocation
127-148일부 kernel allocation은 fallback을 원하지 않거나 허용할 수 없습니다. 지정 node에서 memory를 받거나 그 node에 free memory가 없다는 통지를 확실히 받아야 합니다. Subsystem이 CPU별 memory resource를 할당하는 경우가 보통 그렇습니다.
전형적인 방식은 `numa_node_id()` 또는 `CPU_to_node()`로 current CPU가 연결된 node ID를 얻고 그 node에만 memory를 요청하는 것입니다. 실패하면 subsystem은 자체 fallback path로 돌아갈 수 있습니다. Slab kernel memory allocator가 그 예입니다. 또는 allocation failure 시 subsystem을 disable하거나 enable하지 않을 수 있는데 kernel profiling subsystem이 그 예입니다.
Architecture가 memoryless node를 숨기지 않고 지원한다면 memoryless node에 연결된 CPU는 항상 fallback overhead를 겪거나, 일부 subsystem이 memory 없는 node에서만 할당하려다 초기화에 실패할 수 있습니다. 이를 투명하게 지원하려면 kernel subsystem이 `numa_mem_id()` 또는 `cpu_to_mem()`으로 호출 CPU나 지정 CPU의 local memory node를 찾을 수 있습니다. 이 node가 기본 local-page allocation을 시도하는 바로 그 node입니다.
Some kernel allocations do not want or cannot tolerate this allocation fallback
behavior. Rather they want to be sure they get memory from the specified node
or get notified that the node has no free memory. This is usually the case when
a subsystem allocates per CPU memory resources, for example.
A typical model for making such an allocation is to obtain the node id of the
node to which the "current CPU" is attached using one of the kernel's
numa_node_id() or CPU_to_node() functions and then request memory from only
the node id returned. When such an allocation fails, the requesting subsystem
may revert to its own fallback path. The slab kernel memory allocator is an
example of this. Or, the subsystem may choose to disable or not to enable
itself on allocation failure. The kernel profiling subsystem is an example of
this.
If the architecture supports--does not hide--memoryless nodes, then CPUs
attached to memoryless nodes would always incur the fallback path overhead
or some subsystems would fail to initialize if they attempted to allocated
memory exclusively from a node without memory. To support such
architectures transparently, kernel subsystems can use the numa_mem_id()
or cpu_to_mem() function to locate the "local memory node" for the calling or
specified CPU. Again, this is the same node from which default, local page
allocations will be attempted.
요약·해설
numa.rst:1-148NUMA는 CPU와 memory가 여러 physical cell에 나뉘고 cell 간 거리에 따라 latency와 bandwidth가 달라지는 구조입니다. Linux는 cell을 node로 추상화하고, 각 node의 zone을 NUMA distance 순 zonelist에 배치해 가능한 한 local memory에서 할당한 뒤 가까운 node로 fallback합니다.
Software node는 architecture에 따라 physical cell의 세부 차이를 숨깁니다.
Node-ordered zonelist는 같은 node의 zone을 먼저 보고 remote node를 거리순으로 찾습니다.
Fallback 허용 여부와 memoryless node 지원에 따라 helper가 달라집니다.