Hugetlbfs Reservation

개요와 대상 독자

`Documentation/admin-guide/mm/hugetlbpage.rst`에서 설명하는 huge page는 보통 application이 사용할 수 있도록 미리 할당합니다. VMA가 huge page 사용을 나타내면 page fault 시점에 이 huge page를 task의 address space에 실제로 배치합니다. Page fault 시점에 사용할 huge page가 없으면 task는 `SIGBUS`를 받고 흔히 비정상 종료합니다. Huge page 지원이 추가된 직후, huge page 부족을 `mmap()` 시점에 감지하는 편이 낫다는 판단이 내려졌습니다. Mapping 전체를 감당할 huge page가 충분하지 않으면 `mmap()`을 실패시키자는 발상입니다. 처음에는 `mmap()` 시점에 mapping을 감당할 free huge page가 충분한지 단순히 검사했습니다. Kernel의 많은 코드처럼 이 코드도 시간이 지나며 발전했지만, 기본 개념은 mapping에서 일어날 page fault에 huge page가 반드시 남아 있도록 `mmap()` 시점에 huge page를 '예약'하는 것입니다. 아래 설명은 v4.10 kernel의 huge page reservation 처리 방식을 다룹니다.

이 설명의 주 독자는 hugetlbfs 코드를 수정하는 kernel developer입니다.

=====================
Hugetlbfs Reservation
=====================

Overview
========

Huge pages as described at Documentation/admin-guide/mm/hugetlbpage.rst are
typically preallocated for application use.  These huge pages are instantiated
in a task's address space at page fault time if the VMA indicates huge pages
are to be used.  If no huge page exists at page fault time, the task is sent
a SIGBUS and often dies an unhappy death.  Shortly after huge page support
was added, it was determined that it would be better to detect a shortage
of huge pages at mmap() time.  The idea is that if there were not enough
huge pages to cover the mapping, the mmap() would fail.  This was first
done with a simple check in the code at mmap() time to determine if there
were enough free huge pages to cover the mapping.  Like most things in the
kernel, the code has evolved over time.  However, the basic idea was to
'reserve' huge pages at mmap() time to ensure that huge pages would be
available for page faults in that mapping.  The description below attempts to
describe how huge page reserve processing is done in the v4.10 kernel.


Audience
========
This description is primarily targeted at kernel developers who are modifying
hugetlbfs code.

예약 자료 구조

`resv_huge_pages`는 예약된 huge page의 전역, 정확히는 hstate별 개수입니다. 예약된 huge page는 그것을 예약한 task만 사용할 수 있으므로, 일반적으로 사용할 수 있는 huge page 수는 `free_huge_pages - resv_huge_pages`로 계산합니다.

Reserve map은 다음 구조체로 표현합니다.

		struct resv_map {
			struct kref refs;
			spinlock_t lock;
			struct list_head regions;
			long adds_in_progress;
			struct list_head region_cache;
			long region_cache_count;
		};

System의 각 huge page mapping에는 reserve map이 하나씩 있습니다. `resv_map` 안의 `regions` list는 mapping 내부의 region을 기술하며, region은 다음 구조체로 표현합니다.

		struct file_region {
			struct list_head link;
			long from;
			long to;
		};

`file_region` 구조체의 `from`과 `to` field는 mapping 안의 huge page index입니다. Mapping 종류에 따라 `resv_map`의 region은 해당 범위에 reservation이 있음을 뜻할 수도 있고, 없음을 뜻할 수도 있습니다.

`MAP_PRIVATE` reservation flag는 reservation map pointer의 하위 bit에 저장합니다.

• `HPAGE_RESV_OWNER (1UL << 0)`: 이 task가 mapping에 연결된 reservation의 owner임을 나타냅니다.
• `HPAGE_RESV_UNMAPPED (1UL << 1)`: 원래 이 범위를 mapping하고 reservation을 만든 task가 COW 실패 때문에 이 task, 즉 child에서 page 하나를 unmap했음을 나타냅니다.

`PagePrivate` page flag는 huge page가 해제될 때 huge page reservation을 복원해야 함을 나타냅니다. 자세한 내용은 'Huge page 해제' 구간에서 설명합니다.

The Data Structures
===================

resv_huge_pages
        This is a global (per-hstate) count of reserved huge pages.  Reserved
        huge pages are only available to the task which reserved them.
        Therefore, the number of huge pages generally available is computed
        as (``free_huge_pages - resv_huge_pages``).
Reserve Map
        A reserve map is described by the structure::

                struct resv_map {
                        struct kref refs;
                        spinlock_t lock;
                        struct list_head regions;
                        long adds_in_progress;
                        struct list_head region_cache;
                        long region_cache_count;
                };

        There is one reserve map for each huge page mapping in the system.
        The regions list within the resv_map describes the regions within
        the mapping.  A region is described as::

                struct file_region {
                        struct list_head link;
                        long from;
                        long to;
                };

        The 'from' and 'to' fields of the file region structure are huge page
        indices into the mapping.  Depending on the type of mapping, a
        region in the reserv_map may indicate reservations exist for the
        range, or reservations do not exist.
Flags for MAP_PRIVATE Reservations
        These are stored in the bottom bits of the reservation map pointer.

        ``#define HPAGE_RESV_OWNER    (1UL << 0)``
                Indicates this task is the owner of the reservations
                associated with the mapping.
        ``#define HPAGE_RESV_UNMAPPED (1UL << 1)``
                Indicates task originally mapping this range (and creating
                reserves) has unmapped a page from this task (the child)
                due to a failed COW.
Page Flags
        The PagePrivate page flag is used to indicate that a huge page
        reservation must be restored when the huge page is freed.  More
        details will be discussed in the "Freeing huge pages" section.

Private와 shared mapping의 reserve map 위치

Huge page mapping 또는 segment는 private이거나 shared입니다. Private mapping은 일반적으로 하나의 address space, 즉 task만 사용할 수 있습니다. Shared mapping은 여러 address space, 즉 여러 task에 mapping할 수 있습니다. 두 mapping 종류는 reservation map의 위치와 의미가 크게 다릅니다.

• Private mapping의 reservation map은 VMA 구조체, 구체적으로 `vma->vm_private_data`에 매달립니다. 이 reserve map은 `mmap(MAP_PRIVATE)` mapping을 만들 때 생성합니다.
• Shared mapping의 reservation map은 inode, 구체적으로 `inode->i_mapping->private_data`에 매달립니다. Shared mapping은 항상 hugetlbfs filesystem의 file을 기반으로 하므로 hugetlbfs 코드는 각 inode에 reservation map이 있도록 보장합니다. 따라서 inode를 만들 때 reservation map도 할당합니다.

Reservation Map Location (Private or Shared)
============================================

A huge page mapping or segment is either private or shared.  If private,
it is typically only available to a single address space (task).  If shared,
it can be mapped into multiple address spaces (tasks).  The location and
semantics of the reservation map is significantly different for the two types
of mappings.  Location differences are:

- For private mappings, the reservation map hangs off the VMA structure.
  Specifically, vma->vm_private_data.  This reserve map is created at the
  time the mapping (mmap(MAP_PRIVATE)) is created.
- For shared mappings, the reservation map hangs off the inode.  Specifically,
  inode->i_mapping->private_data.  Since shared mappings are always backed
  by files in the hugetlbfs filesystem, the hugetlbfs code ensures each inode
  contains a reservation map.  As a result, the reservation map is allocated
  when the inode is created.

Reservation 생성

Huge page 기반 shared memory segment를 `shmget(SHM_HUGETLB)`로 만들거나 `mmap(MAP_HUGETLB)`로 mapping을 만들 때 reservation을 생성합니다. 이 operation은 다음 `hugetlb_reserve_pages()` 호출로 이어집니다.

	int hugetlb_reserve_pages(struct inode *inode,
				  long from, long to,
				  struct vm_area_struct *vma,
				  vm_flags_t vm_flags)

`hugetlb_reserve_pages()`가 가장 먼저 하는 일은 `shmget()` 또는 `mmap()` 호출에 `NORESERVE` flag가 지정됐는지 확인하는 것입니다. `NORESERVE`를 지정했다면 reservation을 원하지 않는 것이므로 즉시 반환합니다.

`from`과 `to` argument는 mapping 또는 기반 file 안의 huge page index입니다. `shmget()`에서는 `from`이 항상 0이고 `to`는 segment 또는 mapping 길이에 해당합니다. `mmap()`에서는 offset argument로 기반 file 내부의 offset을 지정할 수 있으며, 이 경우 `from`과 `to`는 그 offset을 반영해 조정된 값입니다.

PRIVATE mapping과 SHARED mapping의 큰 차이 중 하나는 reservation map에서 reservation을 표현하는 방식입니다.

• Shared mapping에서는 reservation map 항목이 해당 page의 reservation이 현재 존재하거나 과거에 존재했음을 뜻합니다. Reservation을 소비해도 reservation map은 바뀌지 않습니다.
• Private mapping에서는 reservation map 항목이 없다는 것이 해당 page의 reservation이 존재함을 뜻합니다. Reservation을 소비할 때 reservation map에 항목을 추가합니다. 따라서 reservation map으로 어떤 reservation이 소비됐는지도 판별할 수 있습니다.

Private mapping에서 `hugetlb_reserve_pages()`는 reservation map을 만들고 VMA 구조체에 연결합니다. 또한 `HPAGE_RESV_OWNER` flag를 설정해 이 VMA가 reservation의 owner임을 표시합니다.

현재 mapping 또는 segment에 필요한 huge page reservation 수를 구하기 위해 reservation map을 조회합니다. Private mapping에서는 항상 `to - from`입니다. Shared mapping에서는 `to - from` 범위에 reservation 일부가 이미 있을 수 있습니다. 처리 방법은 `Reservation Map Modifications` 구간에서 설명합니다.

Mapping에 subpool이 연결돼 있을 수 있습니다. 그렇다면 subpool에 mapping을 위한 공간이 충분한지 확인합니다. Subpool이 mapping에서 사용할 reservation을 따로 확보했을 수도 있습니다. 자세한 내용은 `Subpool Reservations` 구간을 참조하십시오.

Reservation map과 subpool을 확인하면 새로 필요한 reservation 수를 알 수 있습니다. `hugetlb_acct_memory()`를 호출해 요청한 수의 reservation을 확보할 수 있는지 검사하고 실제로 가져옵니다. 이 함수는 surplus page를 잠재적으로 할당하고 그 개수를 조정하는 routine을 호출하지만, 그 내부에서는 reservation을 감당할 free huge page가 충분한지 확인하는 것이 핵심입니다. 충분하다면 전역 reservation 수 `resv_huge_pages`를 대략 다음과 같이 조정합니다.

	if (resv_needed <= (free_huge_pages - resv_huge_pages)
		resv_huge_pages += resv_needed;

이 counter들을 검사하고 조정하는 동안 전역 lock `hugetlb_lock`을 유지합니다.

Free huge page가 충분해 전역 `resv_huge_pages`를 조정했다면 mapping의 reservation map에도 reservation을 반영합니다. Shared mapping에서는 `from`부터 `to`까지의 범위를 포함하는 `file_region`이 존재하게 됩니다. Private mapping에서는 항목이 없는 상태 자체가 reservation의 존재를 뜻하므로 reservation map을 수정하지 않습니다.

`hugetlb_reserve_pages()`가 성공하면 `from`부터 `to`까지 reservation이 존재하도록 전역 reservation 수와 mapping의 reservation map이 필요한 만큼 수정됩니다.

Creating Reservations
=====================
Reservations are created when a huge page backed shared memory segment is
created (shmget(SHM_HUGETLB)) or a mapping is created via mmap(MAP_HUGETLB).
These operations result in a call to the routine hugetlb_reserve_pages()::

        int hugetlb_reserve_pages(struct inode *inode,
                                  long from, long to,
                                  struct vm_area_struct *vma,
                                  vm_flags_t vm_flags)

The first thing hugetlb_reserve_pages() does is check if the NORESERVE
flag was specified in either the shmget() or mmap() call.  If NORESERVE
was specified, then this routine returns immediately as no reservations
are desired.

The arguments 'from' and 'to' are huge page indices into the mapping or
underlying file.  For shmget(), 'from' is always 0 and 'to' corresponds to
the length of the segment/mapping.  For mmap(), the offset argument could
be used to specify the offset into the underlying file.  In such a case,
the 'from' and 'to' arguments have been adjusted by this offset.

One of the big differences between PRIVATE and SHARED mappings is the way
in which reservations are represented in the reservation map.

- For shared mappings, an entry in the reservation map indicates a reservation
  exists or did exist for the corresponding page.  As reservations are
  consumed, the reservation map is not modified.
- For private mappings, the lack of an entry in the reservation map indicates
  a reservation exists for the corresponding page.  As reservations are
  consumed, entries are added to the reservation map.  Therefore, the
  reservation map can also be used to determine which reservations have
  been consumed.

For private mappings, hugetlb_reserve_pages() creates the reservation map and
hangs it off the VMA structure.  In addition, the HPAGE_RESV_OWNER flag is set
to indicate this VMA owns the reservations.

The reservation map is consulted to determine how many huge page reservations
are needed for the current mapping/segment.  For private mappings, this is
always the value (to - from).  However, for shared mappings it is possible that
some reservations may already exist within the range (to - from).  See the
section :ref:`Reservation Map Modifications <resv_map_modifications>`
for details on how this is accomplished.

The mapping may be associated with a subpool.  If so, the subpool is consulted
to ensure there is sufficient space for the mapping.  It is possible that the
subpool has set aside reservations that can be used for the mapping.  See the
section :ref:`Subpool Reservations <sub_pool_resv>` for more details.

After consulting the reservation map and subpool, the number of needed new
reservations is known.  The routine hugetlb_acct_memory() is called to check
for and take the requested number of reservations.  hugetlb_acct_memory()
calls into routines that potentially allocate and adjust surplus page counts.
However, within those routines the code is simply checking to ensure there
are enough free huge pages to accommodate the reservation.  If there are,
the global reservation count resv_huge_pages is adjusted something like the
following::

        if (resv_needed <= (free_huge_pages - resv_huge_pages)
                resv_huge_pages += resv_needed;

Note that the global lock hugetlb_lock is held when checking and adjusting
these counters.

If there were enough free huge pages and the global count resv_huge_pages
was adjusted, then the reservation map associated with the mapping is
modified to reflect the reservations.  In the case of a shared mapping, a
file_region will exist that includes the range 'from' - 'to'.  For private
mappings, no modifications are made to the reservation map as lack of an
entry indicates a reservation exists.

If hugetlb_reserve_pages() was successful, the global reservation count and
reservation map associated with the mapping will be modified as required to
ensure reservations exist for the range 'from' - 'to'.

Reservation 소비와 huge page 할당

Reservation과 연결된 huge page를 할당해 해당 mapping에 실제로 배치할 때 reservation을 소비합니다. 할당은 다음 `alloc_hugetlb_folio()` 안에서 수행합니다.

	struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
				     unsigned long addr, int avoid_reserve)

`alloc_hugetlb_folio()`는 VMA pointer와 virtual address를 받으므로 reservation map을 조회해 reservation이 있는지 판별할 수 있습니다. `avoid_reserve` argument는 지정 주소에 reserve를 따로 마련해 둔 것처럼 보여도 그것을 사용하지 말라는 뜻입니다. 기존 page의 추가 copy를 할당하는 Copy on Write와 Page Migration에서 가장 자주 사용합니다.

Helper `vma_needs_reservation()`을 호출해 mapping, 즉 VMA 안의 해당 주소에 reservation이 있는지 판별합니다. 자세한 동작은 `Reservation Map Helper Routines` 구간에 있습니다.

`vma_needs_reservation()`의 반환값은 일반적으로 0 또는 1입니다. 0이면 해당 주소에 reservation이 있고, 1이면 없습니다. Reservation이 없고 mapping에 subpool이 연결돼 있다면 subpool이 reservation을 보유하는지 확인합니다. Subpool에 reservation이 있으면 이 할당에 하나를 쓸 수 있습니다. 그러나 어떤 경우든 `avoid_reserve`는 reservation 사용을 금지합니다. Reservation의 존재와 사용 가능 여부를 판단한 뒤 `dequeue_huge_page_vma()`를 호출합니다. 이 routine은 reservation과 관련된 두 argument를 받습니다.

• `avoid_reserve`: `alloc_hugetlb_folio()`에 전달된 것과 같은 값입니다.
• `chg`: 형식은 `long`이지만 `dequeue_huge_page_vma()`에는 0 또는 1만 전달합니다. 0은 reservation이 있다는 뜻이고, 1은 reservation이 없어 가능하다면 global free pool에서 page를 가져와야 한다는 뜻입니다. Memory policy와 관련된 예외는 뒤에서 설명합니다.

VMA의 memory policy와 연결된 free list에서 free page를 찾습니다. Page를 찾으면 free list에서 제거하면서 `free_huge_pages`를 줄입니다. 그 page와 연결된 reservation이 있었다면 다음 조정을 수행합니다.

	SetPagePrivate(page);	/* Indicates allocating this page consumed
				 * a reservation, and if an error is
				 * encountered such that the page must be
				 * freed, the reservation will be restored. */
	resv_huge_pages--;	/* Decrement the global reservation count */

VMA의 memory policy를 만족하는 huge page를 찾지 못하면 buddy allocator로 하나를 할당하려고 시도합니다. 이는 이 문서 범위를 벗어나는 surplus huge page와 overcommit 문제로 이어집니다. Surplus page를 할당하더라도 위와 같이 reservation 기준 조정인 `SetPagePrivate(page)`와 `resv_huge_pages--`를 수행합니다.

새 hugetlb folio를 얻은 뒤 page에 연결된 subpool이 있다면 `(folio)->_hugetlb_subpool`에 그 subpool 값을 설정합니다. Folio가 해제될 때 subpool accounting에 사용됩니다.

이어서 `vma_commit_reservation()`을 호출해 reservation 소비에 맞춰 reserve map을 조정합니다. 일반적으로 region map의 `file_region` 구조체가 page를 나타내도록 보장하는 과정입니다. Reservation이 이미 있던 shared mapping은 reserve map 항목이 존재하므로 바뀌지 않습니다. 그러나 reservation이 없던 shared mapping이거나 private mapping이라면 새 항목을 만들어야 합니다.

`alloc_hugetlb_folio()` 시작 부분에서 `vma_needs_reservation()`을 호출한 뒤 folio 할당 후 `vma_commit_reservation()`을 호출하기 전까지 reserve map이 바뀔 수 있습니다. Shared mapping의 같은 page에 `hugetlb_reserve_pages()`가 호출되면 가능합니다. 이때 reservation count와 subpool free page count가 하나씩 어긋납니다. `vma_needs_reservation()`과 `vma_commit_reservation()` 반환값을 비교해 이 드문 race를 알아낼 수 있습니다. Race를 감지하면 subpool과 global reserve count를 보정합니다.

.. _consume_resv:

Consuming Reservations/Allocating a Huge Page
=============================================

Reservations are consumed when huge pages associated with the reservations
are allocated and instantiated in the corresponding mapping.  The allocation
is performed within the routine alloc_hugetlb_folio()::

        struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
                                     unsigned long addr, int avoid_reserve)

alloc_hugetlb_folio is passed a VMA pointer and a virtual address, so it can
consult the reservation map to determine if a reservation exists.  In addition,
alloc_hugetlb_folio takes the argument avoid_reserve which indicates reserves
should not be used even if it appears they have been set aside for the
specified address.  The avoid_reserve argument is most often used in the case
of Copy on Write and Page Migration where additional copies of an existing
page are being allocated.

The helper routine vma_needs_reservation() is called to determine if a
reservation exists for the address within the mapping(vma).  See the section
:ref:`Reservation Map Helper Routines <resv_map_helpers>` for detailed
information on what this routine does.
The value returned from vma_needs_reservation() is generally
0 or 1.  0 if a reservation exists for the address, 1 if no reservation exists.
If a reservation does not exist, and there is a subpool associated with the
mapping the subpool is consulted to determine if it contains reservations.
If the subpool contains reservations, one can be used for this allocation.
However, in every case the avoid_reserve argument overrides the use of
a reservation for the allocation.  After determining whether a reservation
exists and can be used for the allocation, the routine dequeue_huge_page_vma()
is called.  This routine takes two arguments related to reservations:

- avoid_reserve, this is the same value/argument passed to
  alloc_hugetlb_folio().
- chg, even though this argument is of type long only the values 0 or 1 are
  passed to dequeue_huge_page_vma.  If the value is 0, it indicates a
  reservation exists (see the section "Memory Policy and Reservations" for
  possible issues).  If the value is 1, it indicates a reservation does not
  exist and the page must be taken from the global free pool if possible.

The free lists associated with the memory policy of the VMA are searched for
a free page.  If a page is found, the value free_huge_pages is decremented
when the page is removed from the free list.  If there was a reservation
associated with the page, the following adjustments are made::

        SetPagePrivate(page);        /* Indicates allocating this page consumed
                                 * a reservation, and if an error is
                                 * encountered such that the page must be
                                 * freed, the reservation will be restored. */
        resv_huge_pages--;        /* Decrement the global reservation count */

Note, if no huge page can be found that satisfies the VMA's memory policy
an attempt will be made to allocate one using the buddy allocator.  This
brings up the issue of surplus huge pages and overcommit which is beyond
the scope reservations.  Even if a surplus page is allocated, the same
reservation based adjustments as above will be made: SetPagePrivate(page) and
resv_huge_pages--.

After obtaining a new hugetlb folio, (folio)->_hugetlb_subpool is set to the
value of the subpool associated with the page if it exists.  This will be used
for subpool accounting when the folio is freed.

The routine vma_commit_reservation() is then called to adjust the reserve
map based on the consumption of the reservation.  In general, this involves
ensuring the page is represented within a file_region structure of the region
map.  For shared mappings where the reservation was present, an entry
in the reserve map already existed so no change is made.  However, if there
was no reservation in a shared mapping or this was a private mapping a new
entry must be created.

It is possible that the reserve map could have been changed between the call
to vma_needs_reservation() at the beginning of alloc_hugetlb_folio() and the
call to vma_commit_reservation() after the folio was allocated.  This would
be possible if hugetlb_reserve_pages was called for the same page in a shared
mapping.  In such cases, the reservation count and subpool free page count
will be off by one.  This rare condition can be identified by comparing the
return value from vma_needs_reservation and vma_commit_reservation.  If such
a race is detected, the subpool and global reserve counts are adjusted to
compensate.  See the section
:ref:`Reservation Map Helper Routines <resv_map_helpers>` for more
information on these routines.

Huge page 배치와 해제

Huge page를 할당한 뒤 보통 할당 task의 page table에 page를 추가합니다. 그 전에 shared mapping의 page는 page cache에 추가하고 private mapping의 page는 anonymous reverse mapping에 추가합니다. 두 경우 모두 `PagePrivate` flag를 지웁니다. 따라서 실제로 배치된 huge page를 나중에 해제해도 전역 reservation count `resv_huge_pages`는 조정하지 않습니다.

Huge page는 `free_huge_folio()`가 해제합니다. Generic MM code에서 호출하므로 이 함수에는 folio pointer만 전달됩니다. Huge page를 해제할 때 reservation accounting이 필요할 수 있습니다. Page가 reserve를 가진 subpool과 연결됐거나, global reserve count를 복원해야 하는 error path에서 page를 해제하는 경우입니다.

`page->private` field는 page와 연결된 subpool을 가리킵니다. `PagePrivate` flag가 설정돼 있다면 global reserve count를 조정해야 한다는 뜻입니다.

Routine은 먼저 page에 대해 `hugepage_subpool_put_pages()`를 호출합니다. 이 함수가 전달값 1과 다른 0을 반환하면 subpool에 reserve가 연결돼 있고, 새로 free가 된 page를 사용해 subpool reserve 수를 최소 크기 이상으로 유지해야 한다는 뜻입니다. 따라서 이 경우 전역 `resv_huge_pages` counter를 증가시킵니다.

Page에 `PagePrivate` flag가 설정돼 있었다면 전역 `resv_huge_pages` counter는 항상 증가합니다.

Instantiate Huge Pages
======================

After huge page allocation, the page is typically added to the page tables
of the allocating task.  Before this, pages in a shared mapping are added
to the page cache and pages in private mappings are added to an anonymous
reverse mapping.  In both cases, the PagePrivate flag is cleared.  Therefore,
when a huge page that has been instantiated is freed no adjustment is made
to the global reservation count (resv_huge_pages).


Freeing Huge Pages
==================

Huge pages are freed by free_huge_folio().  It is only passed a pointer
to the folio as it is called from the generic MM code.  When a huge page
is freed, reservation accounting may need to be performed.  This would
be the case if the page was associated with a subpool that contained
reserves, or the page is being freed on an error path where a global
reserve count must be restored.

The page->private field points to any subpool associated with the page.
If the PagePrivate flag is set, it indicates the global reserve count should
be adjusted (see the section
:ref:`Consuming Reservations/Allocating a Huge Page <consume_resv>`
for information on how these are set).

The routine first calls hugepage_subpool_put_pages() for the page.  If this
routine returns a value of 0 (which does not equal the value passed 1) it
indicates reserves are associated with the subpool, and this newly free page
must be used to keep the number of subpool reserves above the minimum size.
Therefore, the global resv_huge_pages counter is incremented in this case.

If the PagePrivate flag was set in the page, the global resv_huge_pages counter
will always be incremented.

Subpool reservation

Huge page 크기마다 `struct hstate`가 하나씩 있으며, hstate는 해당 크기의 모든 huge page를 추적합니다. Subpool은 mount된 hugetlbfs filesystem과 연결된 hstate page의 부분 집합입니다.

Hugetlbfs filesystem을 mount할 때 filesystem에 필요한 최소 huge page 수를 나타내는 `min_size` option을 지정할 수 있습니다. 지정하면 `min_size`에 해당하는 huge page 수를 filesystem 전용으로 예약합니다. 이 수는 `struct hugepage_subpool`의 `min_hpages` field에서 추적합니다. Mount 시점에 `hugetlb_acct_memory(min_hpages)`를 호출해 지정한 huge page 수를 예약하며, 예약할 수 없으면 mount가 실패합니다.

Subpool에서 page를 얻거나 되돌릴 때 `hugepage_subpool_get/put_pages()`를 호출합니다. 이 routine들은 모든 subpool accounting을 수행하고 subpool과 연결된 reservation을 추적합니다. Subpool의 'used page' count를 조정할 huge page 수를 전달하며, get은 count를 줄이고 put은 늘립니다. 보통 전달받은 값을 그대로 반환하고, subpool에 page가 충분하지 않으면 error를 반환합니다.

하지만 subpool에 reserve가 연결돼 있으면 전달값보다 작은 값을 반환할 수 있습니다. 이 반환값은 global pool에서 추가로 조정해야 할 page 수입니다. 예를 들어 subpool이 reserved huge page 3개를 가지고 있는데 5개를 요청하면, subpool reservation 3개로 요청 일부를 충족하고 나머지 2개는 global pool에서 가져와야 합니다. 이 사실을 호출자에게 전달하려고 2를 반환하며, 호출자는 global pool에서 추가 page 2개를 얻어야 합니다.

.. _sub_pool_resv:

Subpool Reservations
====================

There is a struct hstate associated with each huge page size.  The hstate
tracks all huge pages of the specified size.  A subpool represents a subset
of pages within a hstate that is associated with a mounted hugetlbfs
filesystem.

When a hugetlbfs filesystem is mounted a min_size option can be specified
which indicates the minimum number of huge pages required by the filesystem.
If this option is specified, the number of huge pages corresponding to
min_size are reserved for use by the filesystem.  This number is tracked in
the min_hpages field of a struct hugepage_subpool.  At mount time,
hugetlb_acct_memory(min_hpages) is called to reserve the specified number of
huge pages.  If they can not be reserved, the mount fails.

The routines hugepage_subpool_get/put_pages() are called when pages are
obtained from or released back to a subpool.  They perform all subpool
accounting, and track any reservations associated with the subpool.
hugepage_subpool_get/put_pages are passed the number of huge pages by which
to adjust the subpool 'used page' count (down for get, up for put).  Normally,
they return the same value that was passed or an error if not enough pages
exist in the subpool.

However, if reserves are associated with the subpool a return value less
than the passed value may be returned.  This return value indicates the
number of additional global pool adjustments which must be made.  For example,
suppose a subpool contains 3 reserved huge pages and someone asks for 5.
The 3 reserved pages associated with the subpool can be used to satisfy part
of the request.  But, 2 pages must be obtained from the global pools.  To
relay this information to the caller, the value 2 is returned.  The caller
is then responsible for attempting to obtain the additional two pages from
the global pools.

COW와 reservation

Shared mapping은 모두 같은 기반 page를 가리키고 사용하므로 COW에서 reservation이 가장 문제가 되는 것은 private mapping입니다. 이 경우 task 두 개가 이전에 할당된 같은 page를 가리킬 수 있습니다. 한 task가 page에 쓰려고 하면 각 task가 자기 page를 가리키도록 새 page를 할당해야 합니다.

원래 page를 할당할 때 그 page의 reservation을 이미 소비했습니다. COW 때문에 새 page를 할당하려 할 때 free huge page가 하나도 없어 할당이 실패할 수 있습니다.

Private mapping을 처음 만들 때 owner의 reservation map pointer에 `HPAGE_RESV_OWNER` bit를 설정해 mapping owner를 기록했습니다. Mapping을 만든 owner가 mapping과 연결된 모든 reservation을 소유합니다. 따라서 write fault가 발생했는데 사용할 page가 없으면 reservation owner인지 아닌지에 따라 다르게 처리합니다.

Fault를 일으킨 task가 owner가 아니면 fault는 실패하고 task는 보통 `SIGBUS`를 받습니다.

Fault를 일으킨 task가 owner라면 원래 reservation을 소유했으므로 성공시켜야 합니다. 이를 위해 non-owner task에서 page를 unmap해 owner task의 reference만 남깁니다. 또한 non-owner task의 reservation map pointer에 `HPAGE_RESV_UNMAPPED` bit를 설정합니다. Non-owner task는 나중에 존재하지 않는 page에서 fault를 일으키면 `SIGBUS`를 받을 수 있지만, 원래 mapping과 reservation의 owner는 예상대로 동작합니다.

COW and Reservations
====================

Since shared mappings all point to and use the same underlying pages, the
biggest reservation concern for COW is private mappings.  In this case,
two tasks can be pointing at the same previously allocated page.  One task
attempts to write to the page, so a new page must be allocated so that each
task points to its own page.

When the page was originally allocated, the reservation for that page was
consumed.  When an attempt to allocate a new page is made as a result of
COW, it is possible that no free huge pages are free and the allocation
will fail.

When the private mapping was originally created, the owner of the mapping
was noted by setting the HPAGE_RESV_OWNER bit in the pointer to the reservation
map of the owner.  Since the owner created the mapping, the owner owns all
the reservations associated with the mapping.  Therefore, when a write fault
occurs and there is no page available, different action is taken for the owner
and non-owner of the reservation.

In the case where the faulting task is not the owner, the fault will fail and
the task will typically receive a SIGBUS.

If the owner is the faulting task, we want it to succeed since it owned the
original reservation.  To accomplish this, the page is unmapped from the
non-owning task.  In this way, the only reference is from the owning task.
In addition, the HPAGE_RESV_UNMAPPED bit is set in the reservation map pointer
of the non-owning task.  The non-owning task may receive a SIGBUS if it later
faults on a non-present page.  But, the original owner of the
mapping/reservation will behave as expected.

Reservation map 수정

다음 low-level routine으로 reservation map을 수정합니다. 보통 직접 호출하지 않고 reservation map helper가 이 routine 중 하나를 호출합니다. Source code `mm/hugetlb.c`에 비교적 자세히 문서화돼 있습니다.

	long region_chg(struct resv_map *resv, long f, long t);
	long region_add(struct resv_map *resv, long f, long t);
	void region_abort(struct resv_map *resv, long f, long t);
	long region_count(struct resv_map *resv, long f, long t);

Reservation map operation은 보통 두 단계로 진행합니다.

• 1. `region_chg()`가 reserve map을 검사해 지정 범위 `[f, t)`에서 현재 표현되지 않은 page 수를 구합니다. 호출자는 operation에 충분한 huge page가 있는지 전역 검사와 할당을 수행합니다.
• 2a. Operation이 성공할 수 있으면 이전 `region_chg()`와 같은 `[f, t)` 범위에 `region_add()`를 호출해 reservation map을 실제로 수정합니다.
• 2b. Operation이 성공할 수 없으면 같은 `[f, t)` 범위에 `region_abort()`를 호출해 operation을 중단합니다.

이것은 2단계 과정입니다. 같은 범위에 `region_chg()`를 먼저 호출했다면 뒤따르는 `region_add()`와 `region_abort()`는 성공이 보장됩니다. `region_chg()`가 후속 operation, 특히 `region_add()` 성공에 필요한 자료 구조를 미리 할당하기 때문입니다.

`region_chg()`는 범위에서 map에 현재 표현되지 않은 page 수를 반환하고, `region_add()`는 map에 추가한 page 수를 반환합니다. 대부분 두 반환값은 같습니다. 하지만 shared mapping에서는 `region_chg()`와 `region_add()` 사이에 reservation map이 바뀔 수 있어 두 값이 달라질 수 있습니다. 그러면 global count와 subpool accounting이 틀렸을 가능성이 높으므로 호출자가 이 조건을 검사하고 적절히 보정해야 합니다.

`region_del()`은 reservation map에서 region을 제거할 때 호출하며, 보통 다음 상황에서 사용합니다.

• Hugetlbfs filesystem의 file을 제거할 때 inode와 reservation map을 해제하기 전에 모든 `file_region` 구조체를 해제합니다. 이때 범위 `[0, LONG_MAX)`를 전달합니다.
• Hugetlbfs file을 truncate할 때 새 file size 뒤의 할당 page를 모두 해제하고 새 EOF 뒤의 `file_region` 항목도 지웁니다. 이때 `[new_end_of_file, LONG_MAX)`를 전달합니다.
• Hugetlbfs file에 hole을 낼 때 file 중간의 huge page를 하나씩 제거하면서 해당 reservation map 항목도 제거합니다. 이때 `[page_idx, page_idx + 1)`을 전달합니다.

`region_del()`은 항상 reservation map에서 제거한 page 수를 반환합니다. 극히 드물게 실패할 수 있는데, hole punch 중 기존 `file_region` 항목을 둘로 나눠야 하지만 새 구조체를 할당하지 못한 경우뿐입니다. 이 error에서는 `-ENOMEM`을 반환합니다. Reservation map에는 page reservation이 있다고 표시되지만 subpool과 global reservation count에는 그 reservation이 반영되지 않는 문제가 생깁니다. `hugetlb_fix_reserve_counts()`를 호출해 삭제하지 못한 reservation map 항목과 counter가 일치하도록 보정합니다.

Private huge page mapping을 unmap할 때 `region_count()`를 호출합니다. Private mapping에서는 reservation map 항목이 없다는 것이 reservation의 존재를 뜻합니다. 따라서 map 항목 수를 세면 소비한 reservation 수와 남은 수를 알 수 있습니다. 남은 수는 `(end - start) - region_count(resv, start, end)`입니다. Mapping이 사라지므로 남은 reservation 수만큼 subpool과 global reservation count를 줄입니다.

.. _resv_map_modifications:

Reservation Map Modifications
=============================

The following low level routines are used to make modifications to a
reservation map.  Typically, these routines are not called directly.  Rather,
a reservation map helper routine is called which calls one of these low level
routines.  These low level routines are fairly well documented in the source
code (mm/hugetlb.c).  These routines are::

        long region_chg(struct resv_map *resv, long f, long t);
        long region_add(struct resv_map *resv, long f, long t);
        void region_abort(struct resv_map *resv, long f, long t);
        long region_count(struct resv_map *resv, long f, long t);

Operations on the reservation map typically involve two operations:

1) region_chg() is called to examine the reserve map and determine how
   many pages in the specified range [f, t) are NOT currently represented.

   The calling code performs global checks and allocations to determine if
   there are enough huge pages for the operation to succeed.

2)
  a) If the operation can succeed, region_add() is called to actually modify
     the reservation map for the same range [f, t) previously passed to
     region_chg().
  b) If the operation can not succeed, region_abort is called for the same
     range [f, t) to abort the operation.

Note that this is a two step process where region_add() and region_abort()
are guaranteed to succeed after a prior call to region_chg() for the same
range.  region_chg() is responsible for pre-allocating any data structures
necessary to ensure the subsequent operations (specifically region_add()))
will succeed.

As mentioned above, region_chg() determines the number of pages in the range
which are NOT currently represented in the map.  This number is returned to
the caller.  region_add() returns the number of pages in the range added to
the map.  In most cases, the return value of region_add() is the same as the
return value of region_chg().  However, in the case of shared mappings it is
possible for changes to the reservation map to be made between the calls to
region_chg() and region_add().  In this case, the return value of region_add()
will not match the return value of region_chg().  It is likely that in such
cases global counts and subpool accounting will be incorrect and in need of
adjustment.  It is the responsibility of the caller to check for this condition
and make the appropriate adjustments.

The routine region_del() is called to remove regions from a reservation map.
It is typically called in the following situations:

- When a file in the hugetlbfs filesystem is being removed, the inode will
  be released and the reservation map freed.  Before freeing the reservation
  map, all the individual file_region structures must be freed.  In this case
  region_del is passed the range [0, LONG_MAX).
- When a hugetlbfs file is being truncated.  In this case, all allocated pages
  after the new file size must be freed.  In addition, any file_region entries
  in the reservation map past the new end of file must be deleted.  In this
  case, region_del is passed the range [new_end_of_file, LONG_MAX).
- When a hole is being punched in a hugetlbfs file.  In this case, huge pages
  are removed from the middle of the file one at a time.  As the pages are
  removed, region_del() is called to remove the corresponding entry from the
  reservation map.  In this case, region_del is passed the range
  [page_idx, page_idx + 1).

In every case, region_del() will return the number of pages removed from the
reservation map.  In VERY rare cases, region_del() can fail.  This can only
happen in the hole punch case where it has to split an existing file_region
entry and can not allocate a new structure.  In this error case, region_del()
will return -ENOMEM.  The problem here is that the reservation map will
indicate that there is a reservation for the page.  However, the subpool and
global reservation counts will not reflect the reservation.  To handle this
situation, the routine hugetlb_fix_reserve_counts() is called to adjust the
counters so that they correspond with the reservation map entry that could
not be deleted.

region_count() is called when unmapping a private huge page mapping.  In
private mappings, the lack of a entry in the reservation map indicates that
a reservation exists.  Therefore, by counting the number of entries in the
reservation map we know how many reservations were consumed and how many are
outstanding (outstanding = (end - start) - region_count(resv, start, end)).
Since the mapping is going away, the subpool and global reservation counts
are decremented by the number of outstanding reservations.

Reservation map helper routine

Reservation map을 조회하고 수정하는 helper routine이 여러 개 있습니다. 이 함수들은 특정 huge page의 reservation만 다루므로 범위 대신 address 하나를 전달하고, 연결된 VMA도 전달합니다. VMA에서 mapping 종류가 private인지 shared인지와 reservation map이 inode 또는 VMA 중 어디에 있는지 판별할 수 있습니다. 이 helper들은 `Reservation Map Modifications` 구간의 기반 routine을 호출하면서 private과 shared mapping에서 map 항목 의미가 반대라는 차이를 처리해 호출자에게 숨깁니다.

	long vma_needs_reservation(struct hstate *h,
				   struct vm_area_struct *vma,
				   unsigned long addr)

`vma_needs_reservation()`은 지정 page에 `region_chg()`를 호출합니다. Reservation이 없으면 1, 있으면 0을 반환합니다.

	long vma_commit_reservation(struct hstate *h,
				    struct vm_area_struct *vma,
				    unsigned long addr)

`vma_commit_reservation()`은 지정 page에 `region_add()`를 호출합니다. `region_chg()` 뒤에 `region_add()`를 호출하듯 먼저 `vma_needs_reservation()`을 호출한 뒤 사용해야 합니다. Page의 reservation 항목을 추가하며, 추가했으면 1, 추가하지 않았으면 0을 반환합니다. 앞선 `vma_needs_reservation()` 반환값과 비교해야 하며 예상하지 못한 차이는 두 호출 사이에 reservation map이 수정됐음을 뜻합니다.

	void vma_end_reservation(struct hstate *h,
				 struct vm_area_struct *vma,
				 unsigned long addr)

`vma_end_reservation()`은 지정 page에 `region_abort()`를 호출합니다. `region_chg()` 뒤의 `region_abort()`처럼 먼저 `vma_needs_reservation()`을 호출한 뒤 사용하며 진행 중인 reservation add operation을 중단하거나 끝냅니다.

	long vma_add_reservation(struct hstate *h,
				 struct vm_area_struct *vma,
				 unsigned long addr)

`vma_add_reservation()`은 error path에서 reservation 정리를 돕는 특수 wrapper이며 `restore_reserve_on_error()`에서만 호출합니다. `vma_needs_reservation()`과 함께 reservation map에 reservation을 추가하려 할 때 사용합니다. Private과 shared mapping의 서로 다른 의미를 고려해, 항목이 있으면 reservation이 있다는 shared mapping에는 `region_add()`를 호출하고 항목이 없으면 reservation이 있다는 private mapping에는 `region_del()`을 호출합니다.

.. _resv_map_helpers:

Reservation Map Helper Routines
===============================

Several helper routines exist to query and modify the reservation maps.
These routines are only interested with reservations for a specific huge
page, so they just pass in an address instead of a range.  In addition,
they pass in the associated VMA.  From the VMA, the type of mapping (private
or shared) and the location of the reservation map (inode or VMA) can be
determined.  These routines simply call the underlying routines described
in the section "Reservation Map Modifications".  However, they do take into
account the 'opposite' meaning of reservation map entries for private and
shared mappings and hide this detail from the caller::

        long vma_needs_reservation(struct hstate *h,
                                   struct vm_area_struct *vma,
                                   unsigned long addr)

This routine calls region_chg() for the specified page.  If no reservation
exists, 1 is returned.  If a reservation exists, 0 is returned::

        long vma_commit_reservation(struct hstate *h,
                                    struct vm_area_struct *vma,
                                    unsigned long addr)

This calls region_add() for the specified page.  As in the case of region_chg
and region_add, this routine is to be called after a previous call to
vma_needs_reservation.  It will add a reservation entry for the page.  It
returns 1 if the reservation was added and 0 if not.  The return value should
be compared with the return value of the previous call to
vma_needs_reservation.  An unexpected difference indicates the reservation
map was modified between calls::

        void vma_end_reservation(struct hstate *h,
                                 struct vm_area_struct *vma,
                                 unsigned long addr)

This calls region_abort() for the specified page.  As in the case of region_chg
and region_abort, this routine is to be called after a previous call to
vma_needs_reservation.  It will abort/end the in progress reservation add
operation::

        long vma_add_reservation(struct hstate *h,
                                 struct vm_area_struct *vma,
                                 unsigned long addr)

This is a special wrapper routine to help facilitate reservation cleanup
on error paths.  It is only called from the routine restore_reserve_on_error().
This routine is used in conjunction with vma_needs_reservation in an attempt
to add a reservation to the reservation map.  It takes into account the
different reservation map semantics for private and shared mappings.  Hence,
region_add is called for shared mappings (as an entry present in the map
indicates a reservation), and region_del is called for private mappings (as
the absence of an entry in the map indicates a reservation).  See the section
"Reservation cleanup in error paths" for more information on what needs to
be done on error paths.

Error path의 reservation 정리

Reservation map 수정은 두 단계로 진행합니다. Page 할당 전에 `vma_needs_reservation()`을 호출하고, 할당이 성공하면 `vma_commit_reservation()`, 실패하면 `vma_end_reservation()`을 호출합니다. Operation 성공 여부에 따라 global 및 subpool reservation count를 조정하면 상태가 일치합니다.

Huge page를 실제로 배치한 뒤에는 `PagePrivate` flag도 지워 나중에 page를 해제할 때 accounting이 올바르게 이뤄지게 합니다.

그러나 huge page를 할당한 뒤 실제로 배치하기 전에 error가 발생하는 경우가 있습니다. Page 할당이 reservation을 소비하고 subpool, reservation map, global count를 이미 조정한 상태입니다. 이 시점, 즉 page를 배치하고 `PagePrivate`를 지우기 전에 page를 해제하면 `free_huge_folio()`가 global reservation count를 늘립니다. 하지만 reservation map에는 reservation을 소비한 것으로 남습니다. 이 불일치는 reserved huge page의 'leak'을 일으킵니다. Global reserve count가 실제보다 커져 미리 할당한 page의 할당을 막습니다.

`restore_reserve_on_error()`가 이 상황을 처리하려고 합니다. 목표는 reservation map을 page 할당 전 상태로 복원하는 것입니다. 그러면 page를 해제한 뒤 reservation map 상태가 global reservation count와 일치합니다.

`restore_reserve_on_error()` 자체도 reservation map 항목을 복원하다 실패할 수 있습니다. 이 경우 page의 `PagePrivate` flag를 지웁니다. Page를 해제할 때 global reserve count가 증가하지 않게 하는 것입니다. Reservation map에는 reservation을 소비한 것처럼 계속 보입니다. 이 주소에 page를 할당할 수는 있지만 원래 의도했던 reserved page는 사용하지 않습니다.

일부 코드, 특히 `userfaultfd`는 `restore_reserve_on_error()`를 호출할 수 없습니다. 이 경우 huge page를 해제할 때 reservation이 leak되지 않도록 `PagePrivate`만 수정합니다.

Reservation Cleanup in Error Paths
==================================

As mentioned in the section
:ref:`Reservation Map Helper Routines <resv_map_helpers>`, reservation
map modifications are performed in two steps.  First vma_needs_reservation
is called before a page is allocated.  If the allocation is successful,
then vma_commit_reservation is called.  If not, vma_end_reservation is called.
Global and subpool reservation counts are adjusted based on success or failure
of the operation and all is well.

Additionally, after a huge page is instantiated the PagePrivate flag is
cleared so that accounting when the page is ultimately freed is correct.

However, there are several instances where errors are encountered after a huge
page is allocated but before it is instantiated.  In this case, the page
allocation has consumed the reservation and made the appropriate subpool,
reservation map and global count adjustments.  If the page is freed at this
time (before instantiation and clearing of PagePrivate), then free_huge_folio
will increment the global reservation count.  However, the reservation map
indicates the reservation was consumed.  This resulting inconsistent state
will cause the 'leak' of a reserved huge page.  The global reserve count will
be  higher than it should and prevent allocation of a pre-allocated page.

The routine restore_reserve_on_error() attempts to handle this situation.  It
is fairly well documented.  The intention of this routine is to restore
the reservation map to the way it was before the page allocation.   In this
way, the state of the reservation map will correspond to the global reservation
count after the page is freed.

The routine restore_reserve_on_error itself may encounter errors while
attempting to restore the reservation map entry.  In this case, it will
simply clear the PagePrivate flag of the page.  In this way, the global
reserve count will not be incremented when the page is freed.  However, the
reservation map will continue to look as though the reservation was consumed.
A page can still be allocated for the address, but it will not use a reserved
page as originally intended.

There is some code (most notably userfaultfd) which can not call
restore_reserve_on_error.  In this case, it simply modifies the PagePrivate
so that a reservation will not be leaked when the huge page is freed.

Reservation과 memory policy

Linux code를 처음 git으로 관리할 때부터 `struct hstate`에는 node별 huge page list가 있었습니다. Reservation 개념은 그보다 뒤에 추가됐습니다. Reservation을 추가할 때 memory policy는 고려하지 않았습니다. Cpuset은 memory policy와 완전히 같지는 않지만, `hugetlb_acct_memory`의 다음 comment가 reservation과 cpuset 또는 memory policy의 상호작용을 요약합니다.

	/*
	 * When cpuset is configured, it breaks the strict hugetlb page
	 * reservation as the accounting is done on a global variable. Such
	 * reservation is completely rubbish in the presence of cpuset because
	 * the reservation is not checked against page availability for the
	 * current cpuset. Application can still potentially OOM'ed by kernel
	 * with lack of free htlb page in cpuset that the task is in.
	 * Attempt to enforce strict accounting with cpuset is almost
	 * impossible (or too ugly) because cpuset is too fluid that
	 * task or memory node can be dynamically moved between cpusets.
	 *
	 * The change of semantics for shared hugetlb mapping with cpuset is
	 * undesirable. However, in order to preserve some of the semantics,
	 * we fall back to check against current free page availability as
	 * a best attempt and hopefully to minimize the impact of changing
	 * semantics that cpuset has.
	 */

Cpuset을 구성하면 accounting이 전역 변수에 이뤄지므로 엄격한 hugetlb page reservation이 깨집니다. 현재 cpuset에서 page를 쓸 수 있는지 reservation과 대조하지 않기 때문에 application은 자신이 속한 cpuset에 free hugetlb page가 부족하면 여전히 kernel OOM을 겪을 수 있습니다. Task나 memory node를 cpuset 사이에서 동적으로 옮길 수 있어 엄격한 accounting을 강제하기도 거의 불가능하거나 지나치게 복잡합니다. Shared hugetlb mapping의 의미가 cpuset 때문에 달라지는 것은 바람직하지 않으므로, 현재 free page 가용성을 최선의 노력으로 검사해 의미 변화의 영향을 줄입니다.

Huge page reservation은 page fault 시점의 예상하지 못한 page 할당 실패, 즉 OOM을 막기 위해 추가됐습니다. 그러나 application이 cpuset이나 memory policy를 사용하면 필요한 node에 huge page가 있다는 보장은 없습니다. Global reservation 수가 충분해도 마찬가지입니다.

Reservations and Memory Policy
==============================
Per-node huge page lists existed in struct hstate when git was first used
to manage Linux code.  The concept of reservations was added some time later.
When reservations were added, no attempt was made to take memory policy
into account.  While cpusets are not exactly the same as memory policy, this
comment in hugetlb_acct_memory sums up the interaction between reservations
and cpusets/memory policy::

        /*
         * When cpuset is configured, it breaks the strict hugetlb page
         * reservation as the accounting is done on a global variable. Such
         * reservation is completely rubbish in the presence of cpuset because
         * the reservation is not checked against page availability for the
         * current cpuset. Application can still potentially OOM'ed by kernel
         * with lack of free htlb page in cpuset that the task is in.
         * Attempt to enforce strict accounting with cpuset is almost
         * impossible (or too ugly) because cpuset is too fluid that
         * task or memory node can be dynamically moved between cpusets.
         *
         * The change of semantics for shared hugetlb mapping with cpuset is
         * undesirable. However, in order to preserve some of the semantics,
         * we fall back to check against current free page availability as
         * a best attempt and hopefully to minimize the impact of changing
         * semantics that cpuset has.
         */

Huge page reservations were added to prevent unexpected page allocation
failures (OOM) at page fault time.  However, if an application makes use
of cpusets or memory policy there is no guarantee that huge pages will be
available on the required nodes.  This is true even if there are a sufficient
number of global reservations.

Hugetlbfs 회귀 검사

가장 완전한 hugetlb test 모음은 `libhugetlbfs` repository에 있습니다. Hugetlb 관련 코드를 수정했다면 `libhugetlbfs` test suite로 regression을 검사하십시오. 새 hugetlb 기능을 추가했다면 적절한 test도 `libhugetlbfs`에 추가하십시오.

Mike Kravetz, 2017년 4월 7일

Hugetlbfs regression testing
============================

The most complete set of hugetlb tests are in the libhugetlbfs repository.
If you modify any hugetlb related code, use the libhugetlbfs test suite
to check for regressions.  In addition, if you add any new hugetlb
functionality, please add appropriate tests to libhugetlbfs.

--
Mike Kravetz, 7 April 2017