anonymous memory를 RAM에서 회수하려면 CPU가 더 이상 그 physical page를 ordinary mapping으로 사용하지 않도록 해야 합니다. 하지만 virtual address를 그냥 비우면 나중 fault에서 내용을 어디서 복원해야 하는지 잃습니다. swap-out은 present PTE 대신 backing entry와 필요한 software 상태를 남깁니다.
Non-present라고 모두 storage swap은 아닙니다
is_swap_pte()는 present가 아니고 empty도 아닌 PTE를 구분합니다. 이 범주에는 ordinary swap뿐 아니라 migration, device memory, hardware poison, PTE marker 같은 특수 encoding도 들어갈 수 있습니다. 이를 전부 swap file read 요청으로 해석해서는 안 됩니다.
swp_entry_to_pte()는 generic type·offset을 architecture별 PTE layout으로 옮기고, pte_to_swp_entry()는 software flag를 구분·정리한 뒤 generic entry를 얻습니다. raw PTE bit pattern과 swp_entry_t.val은 같은 표현이 아닙니다. x86-64에서는 offset 반전 등 L1TF 완화와 관련된 encoding도 있으므로 그 값을 physical address처럼 읽으면 잘못입니다.
encoding에서 type·offset과 별개로 exclusive·soft-dirty·userfaultfd write-protect 같은 상태를 보존합니다. flag가 entry identity에 섞이지 않도록 conversion helper를 사용해야 합니다. offset에 쓸 수 있는 bit 수와 software flag 위치는 architecture·configuration에 따라 확인해야 하는 구현 세부사항입니다.
Backing을 먼저 마련합니다
ordinary swap-backed anon folio가 아직 swap cache에 없다면 v6.12 reclaim은 add_to_swap()으로 slot과 resident cache 연결을 마련합니다. 그다음 try_to_unmap()이 rmap을 따라 관련 VMA와 PTE를 방문합니다. swap slot 없이 PTE부터 교체하면 fault가 data를 찾을 수 없습니다. v6.18의 folio_alloc_swap(folio, gfp)는 cache 연결까지 수행하고 0을 성공으로 반환하므로 이 boolean API와 조건을 바꾸어 읽어서는 안 됩니다.
여러 process가 같은 anon folio를 mapping할 수 있으므로 한 PTE만 바꾸고 끝나지 않습니다. folio_mapped()와 rmap accounting을 이용해 아직 mapping이 남았는지 확인합니다. folio lock 하나가 모든 page-table 경쟁을 없애 주는 것은 아니며, 각 PTE는 해당 page-table lock 아래 재확인해야 합니다.
PTE 교체에는 rollback이 있습니다
try_to_unmap_one()은 page-table walk가 제공하는 PTE와 lock을 이용합니다. locked VMA를 발견하면 ordinary swap-out을 진행하지 않고 mlock 상태를 맞추는 경로가 필요합니다. large folio에서는 현재 PTE가 가리키는 subpage를 찾아 entry offset을 맞춰야 합니다.
old PTE를 원자적으로 clear하면서 기존 dirty 등 상태를 회수합니다. architecture와 TTU 조건에 따라 TLB flush를 즉시 하거나 batch할 수 있지만, stale translation을 통해 사용 중인 memory를 먼저 free해서는 안 됩니다. dirty 흔적은 folio에 반영하여 backing에 보존할 책임을 잃지 않게 합니다.
PTE에 넣을 swap entry의 참조 accounting은 swap_duplicate() 같은 helper로 증가시킵니다. 실패하면 원래 PTE를 복구해야 합니다. anon-exclusive 상태를 swap과 공유 가능한 형태로 전환하는 검사도 실패할 수 있으며, 이 경우 증가시킨 swap 참조까지 돌려놓아야 합니다. PTE만 복원하고 slot count를 그대로 두면 lifetime이 어긋납니다.
Exclusive는 CPU 한 개만 접근한다는 뜻이 아닙니다
anonymous-exclusive 정보는 CoW·GUP·공유 가능성과 연결된 ownership 성격입니다. 이것만 보고 같은 mm의 다른 thread나 CPU가 절대로 접근하지 않는다고 결론 내릴 수 없습니다. TLB batching의 안전성은 mm의 CPU 사용과 PTE 변경 protocol 등 실제 조건으로 판단해야 합니다.
변환이 가능하면 generic swap entry를 PTE encoding으로 만들고 exclusive·soft-dirty·uffd-wp를 반영해 설치합니다. 이어 rmap과 RSS/swap accounting을 조정하고 present mapping이 가지고 있던 folio reference를 돌려놓습니다. cache-held reference는 별도이므로 folio가 여기서 사라지는 것은 아닙니다.
Unmap 성공은 swap-out 완료가 아닙니다
user PTE가 swap entry가 되었어도 folio는 RAM에 있고 dirty할 수 있습니다. fault는 같은 entry를 cache에서 찾아 다시 mapping할 수 있습니다. 반대로 unmap이 일부라도 실패하면 reclaim은 여전히 mapped인 folio를 free하지 않고 보존합니다.
virtual mapping의 복구 경로를 남기는 일, stale CPU translation을 정리하는 일, backing 내용을 쓰는 일, cache와 RAM을 해제하는 일은 서로 다른 단계입니다. 이 구분이 있어야 swap-out과 swap-in이 중간에 겹쳐도 내용을 잃지 않습니다.
x86-64 encoding을 직접 읽어 봅니다
다음은 Linux v6.0의 전체 conversion helper와 x86-64 macro 묶음입니다. generic entry와 architecture entry를 한 단계씩 변환하는 순서를 확인하실 수 있습니다. upstream notice와 GPL-2.0을 유지한 code입니다.
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_SWAPOPS_H
#define _LINUX_SWAPOPS_H
/*
* Official Linux v6.0: include/linux/swapops.h
* https://github.com/torvalds/linux/blob/v6.0/include/linux/swapops.h#L68
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static inline int is_swap_pte(pte_t pte)
{
return !pte_none(pte) && !pte_present(pte);
}
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_SWAPOPS_H
#define _LINUX_SWAPOPS_H
/*
* Official Linux v6.0: include/linux/swapops.h
* https://github.com/torvalds/linux/blob/v6.0/include/linux/swapops.h#L90
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static inline pte_t swp_entry_to_pte(swp_entry_t entry)
{
swp_entry_t arch_entry;
arch_entry = __swp_entry(swp_type(entry), swp_offset(entry));
return __swp_entry_to_pte(arch_entry);
}
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_SWAPOPS_H
#define _LINUX_SWAPOPS_H
/*
* Official Linux v6.0: include/linux/swapops.h
* https://github.com/torvalds/linux/blob/v6.0/include/linux/swapops.h#L77
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static inline swp_entry_t pte_to_swp_entry(pte_t pte)
{
swp_entry_t arch_entry;
pte = pte_swp_clear_flags(pte);
arch_entry = __pte_to_swp_entry(pte);
return swp_entry(__swp_type(arch_entry), __swp_offset(arch_entry));
}
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_SWAPOPS_H
#define _LINUX_SWAPOPS_H
/*
* Official Linux v6.0: include/linux/swapops.h
* https://github.com/torvalds/linux/blob/v6.0/include/linux/swapops.h#L27
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static inline pte_t pte_swp_clear_flags(pte_t pte)
{
if (pte_swp_exclusive(pte))
pte = pte_swp_clear_exclusive(pte);
if (pte_swp_soft_dirty(pte))
pte = pte_swp_clear_soft_dirty(pte);
if (pte_swp_uffd_wp(pte))
pte = pte_swp_clear_uffd_wp(pte);
return pte;
}
v6_0_x86_swap_encoding_macros.h
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _ASM_X86_PGTABLE_64_H
#define _ASM_X86_PGTABLE_64_H
/*
* Official Linux v6.0: arch/x86/include/asm/pgtable_64.h
* https://github.com/torvalds/linux/blob/v6.0/arch/x86/include/asm/pgtable_64.h#L214
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
#define SWP_TYPE_BITS 5
#define SWP_OFFSET_FIRST_BIT (_PAGE_BIT_PROTNONE + 1)
/* We always extract/encode the offset by shifting it all the way up, and then down again */
#define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT+SWP_TYPE_BITS)
#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS)
/* Extract the high bits for type */
#define __swp_type(x) ((x).val >> (64 - SWP_TYPE_BITS))
/* Shift up (to get rid of type), then down to get value */
#define __swp_offset(x) (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT)
/*
* Shift the offset up "too far" by TYPE bits, then down again
* The offset is inverted by a binary not operation to make the high
* physical bits set.
*/
#define __swp_entry(type, offset) ((swp_entry_t) { \
(~(unsigned long)(offset) << SWP_OFFSET_SHIFT >> SWP_TYPE_BITS) \
| ((unsigned long)(type) << (64-SWP_TYPE_BITS)) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) })
#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val((pmd)) })
#define __swp_entry_to_pte(x) ((pte_t) { .pte = (x).val })
#define __swp_entry_to_pmd(x) ((pmd_t) { .pmd = (x).val })
이 x86-64 layout은 high 5 bits에 type을 두고 low 9 bits는 offset 밖에 둡니다. SWP_OFFSET_SHIFT는 14이고, unsigned offset을 반전해 왼쪽으로 14 bit 이동한 뒤 다시 5 bit 내립니다. 따라서 보존되는 offset 정보는 bits 9–58의 50 bits입니다. 중간 shift가 잘라 내는 bit까지 포함해 식을 읽어야 합니다. 이는 이 architecture layout의 계산이며 generic entry의 type shift가 언제나 59라는 뜻은 아닙니다. 반전은 L1TF 완화와 관련되지만 이 식 하나를 모든 speculative-execution 문제의 해결이라고 확대하지 않습니다.
Rmap walk의 lock 계약과 rollback을 확인합니다
아래 v6.0 code에서 try_to_unmap()의 반환형은 void입니다. 각 VMA callback의 bool 결과와 전체 호출 뒤 folio_mapped() 상태를 혼동하면 안 됩니다. rmap_walk_control은 callback, 종료 판정, anon_vma read-lock 획득 방법을 묶습니다. TTU_RMAP_LOCKED가 있으면 caller가 확보한 rmap lock 계약으로 walk합니다.
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/*
* Official Linux v6.0: mm/rmap.c
* https://github.com/torvalds/linux/blob/v6.0/mm/rmap.c#L1812
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
void try_to_unmap(struct folio *folio, enum ttu_flags flags)
{
struct rmap_walk_control rwc = {
.rmap_one = try_to_unmap_one,
.arg = (void *)flags,
.done = page_not_mapped,
.anon_lock = folio_lock_anon_vma_read,
};
if (flags & TTU_RMAP_LOCKED)
rmap_walk_locked(folio, &rwc);
else
rmap_walk(folio, &rwc);
}
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_RMAP_H
#define _LINUX_RMAP_H
/*
* Declarations for Reverse Mapping functions in mm/rmap.c
*/
/*
* Official Linux v6.0: include/linux/rmap.h
* https://github.com/torvalds/linux/blob/v6.0/include/linux/rmap.h#L316
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
struct page_vma_mapped_walk {
unsigned long pfn;
unsigned long nr_pages;
pgoff_t pgoff;
struct vm_area_struct *vma;
unsigned long address;
pmd_t *pmd;
pte_t *pte;
spinlock_t *ptl;
unsigned int flags;
};
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/*
* Official Linux v6.0: mm/rmap.c
* https://github.com/torvalds/linux/blob/v6.0/mm/rmap.c#L1476
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
unsigned long address, void *arg)
{
struct mm_struct *mm = vma->vm_mm;
DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0);
pte_t pteval;
struct page *subpage;
bool anon_exclusive, ret = true;
struct mmu_notifier_range range;
enum ttu_flags flags = (enum ttu_flags)(long)arg;
/*
* When racing against e.g. zap_pte_range() on another cpu,
* in between its ptep_get_and_clear_full() and page_remove_rmap(),
* try_to_unmap() may return before page_mapped() has become false,
* if page table locking is skipped: use TTU_SYNC to wait for that.
*/
if (flags & TTU_SYNC)
pvmw.flags = PVMW_SYNC;
if (flags & TTU_SPLIT_HUGE_PMD)
split_huge_pmd_address(vma, address, false, folio);
/*
* For THP, we have to assume the worse case ie pmd for invalidation.
* For hugetlb, it could be much worse if we need to do pud
* invalidation in the case of pmd sharing.
*
* Note that the folio can not be freed in this function as call of
* try_to_unmap() must hold a reference on the folio.
*/
range.end = vma_address_end(&pvmw);
mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm,
address, range.end);
if (folio_test_hugetlb(folio)) {
/*
* If sharing is possible, start and end will be adjusted
* accordingly.
*/
adjust_range_if_pmd_sharing_possible(vma, &range.start,
&range.end);
}
mmu_notifier_invalidate_range_start(&range);
while (page_vma_mapped_walk(&pvmw)) {
/* Unexpected PMD-mapped THP? */
VM_BUG_ON_FOLIO(!pvmw.pte, folio);
/*
* If the folio is in an mlock()d vma, we must not swap it out.
*/
if (!(flags & TTU_IGNORE_MLOCK) &&
(vma->vm_flags & VM_LOCKED)) {
/* Restore the mlock which got missed */
mlock_vma_folio(folio, vma, false);
page_vma_mapped_walk_done(&pvmw);
ret = false;
break;
}
subpage = folio_page(folio,
pte_pfn(*pvmw.pte) - folio_pfn(folio));
address = pvmw.address;
anon_exclusive = folio_test_anon(folio) &&
PageAnonExclusive(subpage);
if (folio_test_hugetlb(folio)) {
bool anon = folio_test_anon(folio);
/*
* The try_to_unmap() is only passed a hugetlb page
* in the case where the hugetlb page is poisoned.
*/
VM_BUG_ON_PAGE(!PageHWPoison(subpage), subpage);
/*
* huge_pmd_unshare may unmap an entire PMD page.
* There is no way of knowing exactly which PMDs may
* be cached for this mm, so we must flush them all.
* start/end were already adjusted above to cover this
* range.
*/
flush_cache_range(vma, range.start, range.end);
/*
* To call huge_pmd_unshare, i_mmap_rwsem must be
* held in write mode. Caller needs to explicitly
* do this outside rmap routines.
*/
VM_BUG_ON(!anon && !(flags & TTU_RMAP_LOCKED));
if (!anon && huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
flush_tlb_range(vma, range.start, range.end);
mmu_notifier_invalidate_range(mm, range.start,
range.end);
/*
* The ref count of the PMD page was dropped
* which is part of the way map counting
* is done for shared PMDs. Return 'true'
* here. When there is no other sharing,
* huge_pmd_unshare returns false and we will
* unmap the actual page and drop map count
* to zero.
*/
page_vma_mapped_walk_done(&pvmw);
break;
}
pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
} else {
flush_cache_page(vma, address, pte_pfn(*pvmw.pte));
/*
* Nuke the page table entry. When having to clear
* PageAnonExclusive(), we always have to flush.
*/
if (should_defer_flush(mm, flags) && !anon_exclusive) {
/*
* We clear the PTE but do not flush so potentially
* a remote CPU could still be writing to the folio.
* If the entry was previously clean then the
* architecture must guarantee that a clear->dirty
* transition on a cached TLB entry is written through
* and traps if the PTE is unmapped.
*/
pteval = ptep_get_and_clear(mm, address, pvmw.pte);
set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
} else {
pteval = ptep_clear_flush(vma, address, pvmw.pte);
}
}
/*
* Now the pte is cleared. If this pte was uffd-wp armed,
* we may want to replace a none pte with a marker pte if
* it's file-backed, so we don't lose the tracking info.
*/
pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval);
/* Set the dirty flag on the folio now the pte is gone. */
if (pte_dirty(pteval))
folio_mark_dirty(folio);
/* Update high watermark before we lower rss */
update_hiwater_rss(mm);
if (PageHWPoison(subpage) && !(flags & TTU_IGNORE_HWPOISON)) {
pteval = swp_entry_to_pte(make_hwpoison_entry(subpage));
if (folio_test_hugetlb(folio)) {
hugetlb_count_sub(folio_nr_pages(folio), mm);
set_huge_pte_at(mm, address, pvmw.pte, pteval);
} else {
dec_mm_counter(mm, mm_counter(&folio->page));
set_pte_at(mm, address, pvmw.pte, pteval);
}
} else if (pte_unused(pteval) && !userfaultfd_armed(vma)) {
/*
* The guest indicated that the page content is of no
* interest anymore. Simply discard the pte, vmscan
* will take care of the rest.
* A future reference will then fault in a new zero
* page. When userfaultfd is active, we must not drop
* this page though, as its main user (postcopy
* migration) will not expect userfaults on already
* copied pages.
*/
dec_mm_counter(mm, mm_counter(&folio->page));
/* We have to invalidate as we cleared the pte */
mmu_notifier_invalidate_range(mm, address,
address + PAGE_SIZE);
} else if (folio_test_anon(folio)) {
swp_entry_t entry = { .val = page_private(subpage) };
pte_t swp_pte;
/*
* Store the swap location in the pte.
* See handle_pte_fault() ...
*/
if (unlikely(folio_test_swapbacked(folio) !=
folio_test_swapcache(folio))) {
WARN_ON_ONCE(1);
ret = false;
/* We have to invalidate as we cleared the pte */
mmu_notifier_invalidate_range(mm, address,
address + PAGE_SIZE);
page_vma_mapped_walk_done(&pvmw);
break;
}
/* MADV_FREE page check */
if (!folio_test_swapbacked(folio)) {
int ref_count, map_count;
/*
* Synchronize with gup_pte_range():
* - clear PTE; barrier; read refcount
* - inc refcount; barrier; read PTE
*/
smp_mb();
ref_count = folio_ref_count(folio);
map_count = folio_mapcount(folio);
/*
* Order reads for page refcount and dirty flag
* (see comments in __remove_mapping()).
*/
smp_rmb();
/*
* The only page refs must be one from isolation
* plus the rmap(s) (dropped by discard:).
*/
if (ref_count == 1 + map_count &&
!folio_test_dirty(folio)) {
/* Invalidate as we cleared the pte */
mmu_notifier_invalidate_range(mm,
address, address + PAGE_SIZE);
dec_mm_counter(mm, MM_ANONPAGES);
goto discard;
}
/*
* If the folio was redirtied, it cannot be
* discarded. Remap the page to page table.
*/
set_pte_at(mm, address, pvmw.pte, pteval);
folio_set_swapbacked(folio);
ret = false;
page_vma_mapped_walk_done(&pvmw);
break;
}
if (swap_duplicate(entry) < 0) {
set_pte_at(mm, address, pvmw.pte, pteval);
ret = false;
page_vma_mapped_walk_done(&pvmw);
break;
}
if (arch_unmap_one(mm, vma, address, pteval) < 0) {
swap_free(entry);
set_pte_at(mm, address, pvmw.pte, pteval);
ret = false;
page_vma_mapped_walk_done(&pvmw);
break;
}
if (anon_exclusive &&
page_try_share_anon_rmap(subpage)) {
swap_free(entry);
set_pte_at(mm, address, pvmw.pte, pteval);
ret = false;
page_vma_mapped_walk_done(&pvmw);
break;
}
/*
* Note: We *don't* remember if the page was mapped
* exclusively in the swap pte if the architecture
* doesn't support __HAVE_ARCH_PTE_SWP_EXCLUSIVE. In
* that case, swapin code has to re-determine that
* manually and might detect the page as possibly
* shared, for example, if there are other references on
* the page or if the page is under writeback. We made
* sure that there are no GUP pins on the page that
* would rely on it, so for GUP pins this is fine.
*/
if (list_empty(&mm->mmlist)) {
spin_lock(&mmlist_lock);
if (list_empty(&mm->mmlist))
list_add(&mm->mmlist, &init_mm.mmlist);
spin_unlock(&mmlist_lock);
}
dec_mm_counter(mm, MM_ANONPAGES);
inc_mm_counter(mm, MM_SWAPENTS);
swp_pte = swp_entry_to_pte(entry);
if (anon_exclusive)
swp_pte = pte_swp_mkexclusive(swp_pte);
if (pte_soft_dirty(pteval))
swp_pte = pte_swp_mksoft_dirty(swp_pte);
if (pte_uffd_wp(pteval))
swp_pte = pte_swp_mkuffd_wp(swp_pte);
set_pte_at(mm, address, pvmw.pte, swp_pte);
/* Invalidate as we cleared the pte */
mmu_notifier_invalidate_range(mm, address,
address + PAGE_SIZE);
} else {
/*
* This is a locked file-backed folio,
* so it cannot be removed from the page
* cache and replaced by a new folio before
* mmu_notifier_invalidate_range_end, so no
* concurrent thread might update its page table
* to point at a new folio while a device is
* still using this folio.
*
* See Documentation/mm/mmu_notifier.rst
*/
dec_mm_counter(mm, mm_counter_file(&folio->page));
}
discard:
/*
* No need to call mmu_notifier_invalidate_range() it has be
* done above for all cases requiring it to happen under page
* table lock before mmu_notifier_invalidate_range_end()
*
* See Documentation/mm/mmu_notifier.rst
*/
page_remove_rmap(subpage, vma, folio_test_hugetlb(folio));
if (vma->vm_flags & VM_LOCKED)
mlock_page_drain_local();
folio_put(folio);
}
mmu_notifier_invalidate_range_end(&range);
return ret;
}
walk state의 pfn·nr_pages는 확인할 physical 범위이고 pgoff·VMA·address는 mapping 위치를 연결합니다. PMD·PTE pointer와 ptl은 walk의 현재 page-table 상태입니다. reclaim이 방문하는 모든 mm의 mmap_lock을 미리 보유한다고 가정하지 않습니다. anon_vma 또는 file mapping의 rmap lock과 해당 PTE의 PTL을 구별해야 합니다.
TTU_SYNC는 동시 zap이 PTE를 지운 뒤 rmap count를 아직 줄이지 않은 순간까지 동기화하기 위해 walk flag로 전달합니다. TTU_SPLIT_HUGE_PMD는 PMD mapping을 PTE 수준으로 처리할 준비입니다. normal anon 경로에서는 PTE dirty를 folio에 반영하고 swap reference를 증가시킨 뒤 architecture unmap과 anon-exclusive 전환을 검사합니다. 뒤의 두 단계가 실패하면 swap_free()와 원래 PTE 복구를 함께 수행합니다.
notifier의 시작·범위 invalidation·끝도 각 장치가 옛 translation을 계속 사용하지 않도록 하는 protocol입니다. TTU_IGNORE_MLOCK이 없고 locked VMA이면 mlock을 복구하고 그 mapping에서 멈춥니다. MADV_FREE 후보는 ordinary swap entry 설치와 달리 isolation+rmap 외의 reference와 dirty를 barrier 순서로 검사한 뒤 버릴 수 있습니다. 다시 dirty해졌거나 외부 사용이 보이면 PTE를 복구하고 swap-backed로 되돌립니다.
Swap-out을 둘러싼 전체 reclaim 함수
v6.0 shrink_page_list() 전체를 함께 두겠습니다. swap 예약, unmap, pageout, cache 제거만 발췌하면 그 앞의 pin·writeback·reference 보존과 그 뒤의 relock·최종 free 조건이 사라지기 쉽습니다. 이 code의 nr_reclaimed에는 demotion 성공도 포함될 수 있어 전부 system-wide RAM 해제로 읽지 않습니다.
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
/*
* Official Linux v6.0: mm/vmscan.c
* https://github.com/torvalds/linux/blob/v6.0/mm/vmscan.c#L1589
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static unsigned int shrink_page_list(struct list_head *page_list,
struct pglist_data *pgdat,
struct scan_control *sc,
struct reclaim_stat *stat,
bool ignore_references)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
LIST_HEAD(demote_pages);
unsigned int nr_reclaimed = 0;
unsigned int pgactivate = 0;
bool do_demote_pass;
struct swap_iocb *plug = NULL;
memset(stat, 0, sizeof(*stat));
cond_resched();
do_demote_pass = can_demote(pgdat->node_id, sc);
retry:
while (!list_empty(page_list)) {
struct address_space *mapping;
struct folio *folio;
enum page_references references = PAGEREF_RECLAIM;
bool dirty, writeback;
unsigned int nr_pages;
cond_resched();
folio = lru_to_folio(page_list);
list_del(&folio->lru);
if (!folio_trylock(folio))
goto keep;
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
nr_pages = folio_nr_pages(folio);
/* Account the number of base pages */
sc->nr_scanned += nr_pages;
if (unlikely(!folio_evictable(folio)))
goto activate_locked;
if (!sc->may_unmap && folio_mapped(folio))
goto keep_locked;
/*
* The number of dirty pages determines if a node is marked
* reclaim_congested. kswapd will stall and start writing
* folios if the tail of the LRU is all dirty unqueued folios.
*/
folio_check_dirty_writeback(folio, &dirty, &writeback);
if (dirty || writeback)
stat->nr_dirty += nr_pages;
if (dirty && !writeback)
stat->nr_unqueued_dirty += nr_pages;
/*
* Treat this folio as congested if folios are cycling
* through the LRU so quickly that the folios marked
* for immediate reclaim are making it to the end of
* the LRU a second time.
*/
if (writeback && folio_test_reclaim(folio))
stat->nr_congested += nr_pages;
/*
* If a folio at the tail of the LRU is under writeback, there
* are three cases to consider.
*
* 1) If reclaim is encountering an excessive number
* of folios under writeback and this folio has both
* the writeback and reclaim flags set, then it
* indicates that folios are being queued for I/O but
* are being recycled through the LRU before the I/O
* can complete. Waiting on the folio itself risks an
* indefinite stall if it is impossible to writeback
* the folio due to I/O error or disconnected storage
* so instead note that the LRU is being scanned too
* quickly and the caller can stall after the folio
* list has been processed.
*
* 2) Global or new memcg reclaim encounters a folio that is
* not marked for immediate reclaim, or the caller does not
* have __GFP_FS (or __GFP_IO if it's simply going to swap,
* not to fs). In this case mark the folio for immediate
* reclaim and continue scanning.
*
* Require may_enter_fs() because we would wait on fs, which
* may not have submitted I/O yet. And the loop driver might
* enter reclaim, and deadlock if it waits on a folio for
* which it is needed to do the write (loop masks off
* __GFP_IO|__GFP_FS for this reason); but more thought
* would probably show more reasons.
*
* 3) Legacy memcg encounters a folio that already has the
* reclaim flag set. memcg does not have any dirty folio
* throttling so we could easily OOM just because too many
* folios are in writeback and there is nothing else to
* reclaim. Wait for the writeback to complete.
*
* In cases 1) and 2) we activate the folios to get them out of
* the way while we continue scanning for clean folios on the
* inactive list and refilling from the active list. The
* observation here is that waiting for disk writes is more
* expensive than potentially causing reloads down the line.
* Since they're marked for immediate reclaim, they won't put
* memory pressure on the cache working set any longer than it
* takes to write them to disk.
*/
if (folio_test_writeback(folio)) {
/* Case 1 above */
if (current_is_kswapd() &&
folio_test_reclaim(folio) &&
test_bit(PGDAT_WRITEBACK, &pgdat->flags)) {
stat->nr_immediate += nr_pages;
goto activate_locked;
/* Case 2 above */
} else if (writeback_throttling_sane(sc) ||
!folio_test_reclaim(folio) ||
!may_enter_fs(folio, sc->gfp_mask)) {
/*
* This is slightly racy -
* folio_end_writeback() might have
* just cleared the reclaim flag, then
* setting the reclaim flag here ends up
* interpreted as the readahead flag - but
* that does not matter enough to care.
* What we do want is for this folio to
* have the reclaim flag set next time
* memcg reclaim reaches the tests above,
* so it will then wait for writeback to
* avoid OOM; and it's also appropriate
* in global reclaim.
*/
folio_set_reclaim(folio);
stat->nr_writeback += nr_pages;
goto activate_locked;
/* Case 3 above */
} else {
folio_unlock(folio);
folio_wait_writeback(folio);
/* then go back and try same folio again */
list_add_tail(&folio->lru, page_list);
continue;
}
}
if (!ignore_references)
references = folio_check_references(folio, sc);
switch (references) {
case PAGEREF_ACTIVATE:
goto activate_locked;
case PAGEREF_KEEP:
stat->nr_ref_keep += nr_pages;
goto keep_locked;
case PAGEREF_RECLAIM:
case PAGEREF_RECLAIM_CLEAN:
; /* try to reclaim the folio below */
}
/*
* Before reclaiming the folio, try to relocate
* its contents to another node.
*/
if (do_demote_pass &&
(thp_migration_supported() || !folio_test_large(folio))) {
list_add(&folio->lru, &demote_pages);
folio_unlock(folio);
continue;
}
/*
* Anonymous process memory has backing store?
* Try to allocate it some swap space here.
* Lazyfree folio could be freed directly
*/
if (folio_test_anon(folio) && folio_test_swapbacked(folio)) {
if (!folio_test_swapcache(folio)) {
if (!(sc->gfp_mask & __GFP_IO))
goto keep_locked;
if (folio_maybe_dma_pinned(folio))
goto keep_locked;
if (folio_test_large(folio)) {
/* cannot split folio, skip it */
if (!can_split_folio(folio, NULL))
goto activate_locked;
/*
* Split folios without a PMD map right
* away. Chances are some or all of the
* tail pages can be freed without IO.
*/
if (!folio_entire_mapcount(folio) &&
split_folio_to_list(folio,
page_list))
goto activate_locked;
}
if (!add_to_swap(folio)) {
if (!folio_test_large(folio))
goto activate_locked_split;
/* Fallback to swap normal pages */
if (split_folio_to_list(folio,
page_list))
goto activate_locked;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
count_vm_event(THP_SWPOUT_FALLBACK);
#endif
if (!add_to_swap(folio))
goto activate_locked_split;
}
}
} else if (folio_test_swapbacked(folio) &&
folio_test_large(folio)) {
/* Split shmem folio */
if (split_folio_to_list(folio, page_list))
goto keep_locked;
}
/*
* If the folio was split above, the tail pages will make
* their own pass through this function and be accounted
* then.
*/
if ((nr_pages > 1) && !folio_test_large(folio)) {
sc->nr_scanned -= (nr_pages - 1);
nr_pages = 1;
}
/*
* The folio is mapped into the page tables of one or more
* processes. Try to unmap it here.
*/
if (folio_mapped(folio)) {
enum ttu_flags flags = TTU_BATCH_FLUSH;
bool was_swapbacked = folio_test_swapbacked(folio);
if (folio_test_pmd_mappable(folio))
flags |= TTU_SPLIT_HUGE_PMD;
try_to_unmap(folio, flags);
if (folio_mapped(folio)) {
stat->nr_unmap_fail += nr_pages;
if (!was_swapbacked &&
folio_test_swapbacked(folio))
stat->nr_lazyfree_fail += nr_pages;
goto activate_locked;
}
}
mapping = folio_mapping(folio);
if (folio_test_dirty(folio)) {
/*
* Only kswapd can writeback filesystem folios
* to avoid risk of stack overflow. But avoid
* injecting inefficient single-folio I/O into
* flusher writeback as much as possible: only
* write folios when we've encountered many
* dirty folios, and when we've already scanned
* the rest of the LRU for clean folios and see
* the same dirty folios again (with the reclaim
* flag set).
*/
if (folio_is_file_lru(folio) &&
(!current_is_kswapd() ||
!folio_test_reclaim(folio) ||
!test_bit(PGDAT_DIRTY, &pgdat->flags))) {
/*
* Immediately reclaim when written back.
* Similar in principle to deactivate_page()
* except we already have the folio isolated
* and know it's dirty
*/
node_stat_mod_folio(folio, NR_VMSCAN_IMMEDIATE,
nr_pages);
folio_set_reclaim(folio);
goto activate_locked;
}
if (references == PAGEREF_RECLAIM_CLEAN)
goto keep_locked;
if (!may_enter_fs(folio, sc->gfp_mask))
goto keep_locked;
if (!sc->may_writepage)
goto keep_locked;
/*
* Folio is dirty. Flush the TLB if a writable entry
* potentially exists to avoid CPU writes after I/O
* starts and then write it out here.
*/
try_to_unmap_flush_dirty();
switch (pageout(folio, mapping, &plug)) {
case PAGE_KEEP:
goto keep_locked;
case PAGE_ACTIVATE:
goto activate_locked;
case PAGE_SUCCESS:
stat->nr_pageout += nr_pages;
if (folio_test_writeback(folio))
goto keep;
if (folio_test_dirty(folio))
goto keep;
/*
* A synchronous write - probably a ramdisk. Go
* ahead and try to reclaim the folio.
*/
if (!folio_trylock(folio))
goto keep;
if (folio_test_dirty(folio) ||
folio_test_writeback(folio))
goto keep_locked;
mapping = folio_mapping(folio);
fallthrough;
case PAGE_CLEAN:
; /* try to free the folio below */
}
}
/*
* If the folio has buffers, try to free the buffer
* mappings associated with this folio. If we succeed
* we try to free the folio as well.
*
* We do this even if the folio is dirty.
* filemap_release_folio() does not perform I/O, but it
* is possible for a folio to have the dirty flag set,
* but it is actually clean (all its buffers are clean).
* This happens if the buffers were written out directly,
* with submit_bh(). ext3 will do this, as well as
* the blockdev mapping. filemap_release_folio() will
* discover that cleanness and will drop the buffers
* and mark the folio clean - it can be freed.
*
* Rarely, folios can have buffers and no ->mapping.
* These are the folios which were not successfully
* invalidated in truncate_cleanup_folio(). We try to
* drop those buffers here and if that worked, and the
* folio is no longer mapped into process address space
* (refcount == 1) it can be freed. Otherwise, leave
* the folio on the LRU so it is swappable.
*/
if (folio_has_private(folio)) {
if (!filemap_release_folio(folio, sc->gfp_mask))
goto activate_locked;
if (!mapping && folio_ref_count(folio) == 1) {
folio_unlock(folio);
if (folio_put_testzero(folio))
goto free_it;
else {
/*
* rare race with speculative reference.
* the speculative reference will free
* this folio shortly, so we may
* increment nr_reclaimed here (and
* leave it off the LRU).
*/
nr_reclaimed += nr_pages;
continue;
}
}
}
if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) {
/* follow __remove_mapping for reference */
if (!folio_ref_freeze(folio, 1))
goto keep_locked;
/*
* The folio has only one reference left, which is
* from the isolation. After the caller puts the
* folio back on the lru and drops the reference, the
* folio will be freed anyway. It doesn't matter
* which lru it goes on. So we don't bother checking
* the dirty flag here.
*/
count_vm_events(PGLAZYFREED, nr_pages);
count_memcg_folio_events(folio, PGLAZYFREED, nr_pages);
} else if (!mapping || !__remove_mapping(mapping, folio, true,
sc->target_mem_cgroup))
goto keep_locked;
folio_unlock(folio);
free_it:
/*
* Folio may get swapped out as a whole, need to account
* all pages in it.
*/
nr_reclaimed += nr_pages;
/*
* Is there need to periodically free_page_list? It would
* appear not as the counts should be low
*/
if (unlikely(folio_test_large(folio)))
destroy_large_folio(folio);
else
list_add(&folio->lru, &free_pages);
continue;
activate_locked_split:
/*
* The tail pages that are failed to add into swap cache
* reach here. Fixup nr_scanned and nr_pages.
*/
if (nr_pages > 1) {
sc->nr_scanned -= (nr_pages - 1);
nr_pages = 1;
}
activate_locked:
/* Not a candidate for swapping, so reclaim swap space. */
if (folio_test_swapcache(folio) &&
(mem_cgroup_swap_full(&folio->page) ||
folio_test_mlocked(folio)))
try_to_free_swap(&folio->page);
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
if (!folio_test_mlocked(folio)) {
int type = folio_is_file_lru(folio);
folio_set_active(folio);
stat->nr_activate[type] += nr_pages;
count_memcg_folio_events(folio, PGACTIVATE, nr_pages);
}
keep_locked:
folio_unlock(folio);
keep:
list_add(&folio->lru, &ret_pages);
VM_BUG_ON_FOLIO(folio_test_lru(folio) ||
folio_test_unevictable(folio), folio);
}
/* 'page_list' is always empty here */
/* Migrate folios selected for demotion */
nr_reclaimed += demote_page_list(&demote_pages, pgdat);
/* Folios that could not be demoted are still in @demote_pages */
if (!list_empty(&demote_pages)) {
/* Folios which weren't demoted go back on @page_list for retry: */
list_splice_init(&demote_pages, page_list);
do_demote_pass = false;
goto retry;
}
pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
mem_cgroup_uncharge_list(&free_pages);
try_to_unmap_flush();
free_unref_page_list(&free_pages);
list_splice(&ret_pages, page_list);
count_vm_events(PGACTIVATE, pgactivate);
if (plug)
swap_write_unplug(plug);
return nr_reclaimed;
}
참고 자료
- Lorenzo Stoakes, The Linux Memory Manager, Early Access 2025, Chapter 12, Page Table Mappings / Setting Page Table Swap Entries.
- Linux v6.12 mm/rmap.c:
try_to_unmap(),try_to_unmap_one(). - Linux v6.12 include/linux/swapops.h, x86 pgtable_64.h.