reclaim이나 compaction에서 drain이라는 이름을 보면 무엇을 비우는지 먼저 확인해야 합니다. LRU operation batch의 pending 작업을 실행하는 것과, PCP의 free page를 buddy로 돌려보내는 것은 전혀 다른 일입니다. 둘 다 CPU-local 상태를 정리하지만 소유한 object의 상태가 다릅니다.
LRU drain은 pending 작업을 shared list에 반영합니다
batch가 full이 되면 해당 callback으로 자연스럽게 drain합니다. 하지만 reclaim을 시작하거나 migration을 준비할 때에는 아직 덜 찬 batch에도 처리할 folio가 남지 않도록 명시적으로 drain할 수 있습니다. lru_add_drain()은 이름과 달리 addition만이 아니라 여러 pending LRU 작업과 mlock batch를 정리하는 interface입니다.
일반 CPU batch의 addition·rotation·deactivation·lazy-free·activation은 각각 맞는 callback을 실행해야 합니다. 하나의 list로 합쳐 모두 같은 operation을 적용하면 안 됩니다. local mlock batch는 자신의 local lock 아래 처리하고, offline CPU의 remote drain은 그 CPU가 더 이상 producer로 실행되지 않는다는 조건을 이용합니다. v6.18의 mlock batch helper 이름은 mlock_folio_batch(), local drain은 mlock_drain_local()이며, 옛 pagevec 이름과 구분해서 추적해야 합니다.
data_race() 표시는 특정 관찰에 허용된 경쟁을 KCSAN 등에 알리는 annotation입니다. 이것만으로 synchronization이 생기거나 어떤 race든 안전해지는 것은 아닙니다. pending 여부를 가볍게 보는 검사와 실제 batch를 수정하는 lock 구간을 구분해야 합니다.
PCP의 page는 이미 free입니다
PCP는 allocator가 빠르게 재사용할 free page를 CPU-local하게 유지하는 구조입니다. drain_local_pages()나 allocator의 global PCP drain은 이런 page를 buddy free list로 되돌립니다. higher-order coalescing이나 다른 CPU의 allocation에 도움이 될 수 있습니다.
반면 LRU batch의 folio는 user memory나 cache object일 수 있습니다. LRU drain만으로 그것이 free가 되는 것은 아닙니다. callback이 list 상태를 바꾸고 pending reference를 반환할 뿐이며, 실제 memory 해제는 마지막 reference와 별도의 reclaim 조건에 달려 있습니다. compaction이 두 drain을 함께 호출하는 이유도 서로 다른 종류의 숨은 CPU-local 상태를 정리해야 하기 때문입니다.
모든 CPU를 drain할 때에는 생산과 경쟁합니다
global LRU drain은 각 CPU에 pending 작업이 있는지 cpu_needs_drain() 등으로 판단하고 필요한 CPU의 작업을 실행·완료시킵니다. 한 시점의 nr != 0 검사만으로 모든 경쟁을 해결할 수는 없습니다. 검사 직후 다른 CPU가 새 작업을 enqueue할 수 있고 CPU hotplug도 발생할 수 있기 때문입니다.
따라서 global drain의 synchronization을 생략한 개념도와 실제 구현을 동일하게 보면 안 됩니다. 호출자가 무엇을 보장받는지, drain 중 새 producer를 허용하는지, remote work 완료를 기다리는지를 함께 읽어야 합니다. 여러 CPU에 작업을 요청하고 즉시 반환하는 것만으로 global completion이 되지 않습니다.
고전 __lru_add_drain_all()은 먼저 memory barrier로 앞선 batch counter store의 관찰 순서를 맞추고 drain generation을 acquire-load합니다. mutex를 잡은 뒤 다른 호출이 더 새로운 generation을 이미 처리했고 강제 drain이 아니라면 중복 작업을 피합니다. 진행하는 호출은 generation을 증가시키고 다시 barrier를 둔 다음 CPU별 counter를 검사합니다. 이 순서를 검사 loop 뒤로 옮기면 다른 호출이 자기 enqueue까지 처리됐다고 오해하고 빠져나갈 수 있습니다.
필요한 CPU에는 mm_percpu_wq의 per-CPU work를 queue하고, 선택한 CPU mask를 따라 flush_work()로 완료를 기다립니다. work를 분산 실행한다는 뜻에서 비동기 작업을 사용하지만, 이 함수 전체가 '요청만 넣고 즉시 반환'하는 API는 아닙니다. v6.18은 이 경합 구간에 들어가기 전 자기 CPU의 lru_add_drain()도 먼저 수행합니다.
CPU별 drain은 addition, IRQ-protected rotation, file deactivation, 일반 deactivation, lazy-free, activation을 각각 맞는 callback으로 처리합니다. lru_add_and_bh_lrus_drain()은 buffer-head LRU invalidation도 더하고, lru_add_drain_cpu_zone()은 지정 zone의 local PCP를 buddy로 돌리는 작업도 더합니다. 후자를 모든 CPU·모든 zone의 PCP drain이라고 넓혀 읽지 않아야 합니다.
LRU cache disable은 중첩 가능한 상태입니다
memory offlining이나 일부 migration·mempolicy 작업에서는 folio가 batch에 숨어 있는 상태를 허용하지 않으려 합니다. lru_cache_disable()는 disable count를 올리고 필요한 global drain을 수행하여 이후 작업이 즉시 반영되도록 합니다. 대응하는 enable은 count를 내려 상태를 복구합니다.
count인 이유는 caller가 중첩될 수 있기 때문입니다. 한 caller가 끝났다고 다른 caller의 disable 요구까지 해제하면 안 됩니다. enable 시 count를 증가시키는 것도 아닙니다. lru_cache_disabled()는 이 상태를 보아 enqueue 이후 즉시 drain하도록 합니다.
disable count를 먼저 올렸다고 이미 'enabled' 값을 읽고 실행 중인 producer가 사라지지는 않습니다. synchronize_rcu_expedited()는 preemption-disabled 구간이나 PREEMPT_RT의 관련 RCU-protected 구간에 있던 reader가 지나가도록 기다립니다. 그 뒤 강제 global drain을 수행해야 이전 정책을 본 producer의 pending 항목까지 정리할 수 있습니다. SMP에서는 __lru_add_drain_all(true), UP에서는 local drain 경로를 사용합니다.
memory offlining의 offline_pages(), migration·contiguous allocation 준비, mbind()의 migration 관련 경로는 이런 안정된 LRU 가시성이 필요한 caller입니다. 정상적인 매 allocation마다 전역 batching을 끄는 방식은 아닙니다. 대응하는 enable은 count를 원자적으로 하나 줄일 뿐이므로 모든 disable caller가 짝을 지켜야 합니다.
이 정책은 file data cache를 끄는 기능이 아닙니다. 끄는 대상은 LRU operation의 CPU-local batching입니다. cache hit율이나 file contents 자체를 없애는 sysctl과 같은 것으로 해석해서는 안 됩니다. 잠깐의 batching 중단으로 얻는 것은 page의 위치와 상태를 더 확실히 관찰·이동할 수 있다는 보장입니다.
전체 code로 ownership과 종료 경로를 확인합니다
고전 v6.0의 CPU-local drain, global work 완료, cache disable/enable을 전체 정의로 연결했습니다. mlock local drain과 offline CPU remote drain도 실제 전제조건이 다르므로 별도로 보아야 합니다.
v6.0 lru_add_drain_cpu
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/swap.c:620-652.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/swap.c#L620
* License text: GPL-2.0.txt (in this directory).
*/
void lru_add_drain_cpu(int cpu)
{
struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
struct folio_batch *fbatch = &fbatches->lru_add;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_add_fn);
fbatch = &per_cpu(lru_rotate.fbatch, cpu);
/* Disabling interrupts below acts as a compiler barrier. */
if (data_race(folio_batch_count(fbatch))) {
unsigned long flags;
/* No harm done if a racing interrupt already did this */
local_lock_irqsave(&lru_rotate.lock, flags);
folio_batch_move_lru(fbatch, lru_move_tail_fn);
local_unlock_irqrestore(&lru_rotate.lock, flags);
}
fbatch = &fbatches->lru_deactivate_file;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_deactivate_file_fn);
fbatch = &fbatches->lru_deactivate;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_deactivate_fn);
fbatch = &fbatches->lru_lazyfree;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_lazyfree_fn);
folio_activate_drain(cpu);
}
v6.0 lru_add_drain
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/swap.c:727-733.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/swap.c#L727
* License text: GPL-2.0.txt (in this directory).
*/
void lru_add_drain(void)
{
local_lock(&cpu_fbatches.lock);
lru_add_drain_cpu(smp_processor_id());
local_unlock(&cpu_fbatches.lock);
mlock_page_drain_local();
}
v6.0 lru_add_and_bh_lrus_drain
v6_0_lru_add_and_bh_lrus_drain.c
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/swap.c:741-748.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/swap.c#L741
* License text: GPL-2.0.txt (in this directory).
*/
static void lru_add_and_bh_lrus_drain(void)
{
local_lock(&cpu_fbatches.lock);
lru_add_drain_cpu(smp_processor_id());
local_unlock(&cpu_fbatches.lock);
invalidate_bh_lrus_cpu();
mlock_page_drain_local();
}
v6.0 lru_add_drain_cpu_zone
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/swap.c:750-757.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/swap.c#L750
* License text: GPL-2.0.txt (in this directory).
*/
void lru_add_drain_cpu_zone(struct zone *zone)
{
local_lock(&cpu_fbatches.lock);
lru_add_drain_cpu(smp_processor_id());
drain_local_pages(zone);
local_unlock(&cpu_fbatches.lock);
mlock_page_drain_local();
}
v6.0 __lru_add_drain_all
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/swap.c:790-879.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/swap.c#L790
* License text: GPL-2.0.txt (in this directory).
*/
static inline void __lru_add_drain_all(bool force_all_cpus)
{
/*
* lru_drain_gen - Global pages generation number
*
* (A) Definition: global lru_drain_gen = x implies that all generations
* 0 < n <= x are already *scheduled* for draining.
*
* This is an optimization for the highly-contended use case where a
* user space workload keeps constantly generating a flow of pages for
* each CPU.
*/
static unsigned int lru_drain_gen;
static struct cpumask has_work;
static DEFINE_MUTEX(lock);
unsigned cpu, this_gen;
/*
* Make sure nobody triggers this path before mm_percpu_wq is fully
* initialized.
*/
if (WARN_ON(!mm_percpu_wq))
return;
/*
* Guarantee folio_batch counter stores visible by this CPU
* are visible to other CPUs before loading the current drain
* generation.
*/
smp_mb();
/*
* (B) Locally cache global LRU draining generation number
*
* The read barrier ensures that the counter is loaded before the mutex
* is taken. It pairs with smp_mb() inside the mutex critical section
* at (D).
*/
this_gen = smp_load_acquire(&lru_drain_gen);
mutex_lock(&lock);
/*
* (C) Exit the draining operation if a newer generation, from another
* lru_add_drain_all(), was already scheduled for draining. Check (A).
*/
if (unlikely(this_gen != lru_drain_gen && !force_all_cpus))
goto done;
/*
* (D) Increment global generation number
*
* Pairs with smp_load_acquire() at (B), outside of the critical
* section. Use a full memory barrier to guarantee that the
* new global drain generation number is stored before loading
* folio_batch counters.
*
* This pairing must be done here, before the for_each_online_cpu loop
* below which drains the page vectors.
*
* Let x, y, and z represent some system CPU numbers, where x < y < z.
* Assume CPU #z is in the middle of the for_each_online_cpu loop
* below and has already reached CPU #y's per-cpu data. CPU #x comes
* along, adds some pages to its per-cpu vectors, then calls
* lru_add_drain_all().
*
* If the paired barrier is done at any later step, e.g. after the
* loop, CPU #x will just exit at (C) and miss flushing out all of its
* added pages.
*/
WRITE_ONCE(lru_drain_gen, lru_drain_gen + 1);
smp_mb();
cpumask_clear(&has_work);
for_each_online_cpu(cpu) {
struct work_struct *work = &per_cpu(lru_add_drain_work, cpu);
if (cpu_needs_drain(cpu)) {
INIT_WORK(work, lru_add_drain_per_cpu);
queue_work_on(cpu, mm_percpu_wq, work);
__cpumask_set_cpu(cpu, &has_work);
}
}
for_each_cpu(cpu, &has_work)
flush_work(&per_cpu(lru_add_drain_work, cpu));
done:
mutex_unlock(&lock);
}
v6.0 cpu_needs_drain
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/swap.c:768-781.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/swap.c#L768
* License text: GPL-2.0.txt (in this directory).
*/
static bool cpu_needs_drain(unsigned int cpu)
{
struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
/* Check these in order of likelihood that they're not zero */
return folio_batch_count(&fbatches->lru_add) ||
data_race(folio_batch_count(&per_cpu(lru_rotate.fbatch, cpu))) ||
folio_batch_count(&fbatches->lru_deactivate_file) ||
folio_batch_count(&fbatches->lru_deactivate) ||
folio_batch_count(&fbatches->lru_lazyfree) ||
folio_batch_count(&fbatches->activate) ||
need_mlock_page_drain(cpu) ||
has_bh_in_lru(cpu, NULL);
}
v6.0 lru_cache_disable
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/swap.c:902-924.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/swap.c#L902
* License text: GPL-2.0.txt (in this directory).
*/
void lru_cache_disable(void)
{
atomic_inc(&lru_disable_count);
/*
* Readers of lru_disable_count are protected by either disabling
* preemption or rcu_read_lock:
*
* preempt_disable, local_irq_disable [bh_lru_lock()]
* rcu_read_lock [rt_spin_lock CONFIG_PREEMPT_RT]
* preempt_disable [local_lock !CONFIG_PREEMPT_RT]
*
* Since v5.1 kernel, synchronize_rcu() is guaranteed to wait on
* preempt_disable() regions of code. So any CPU which sees
* lru_disable_count = 0 will have exited the critical
* section when synchronize_rcu() returns.
*/
synchronize_rcu_expedited();
#ifdef CONFIG_SMP
__lru_add_drain_all(true);
#else
lru_add_and_bh_lrus_drain();
#endif
}
v6.0 mlock_page_drain_local
// SPDX-License-Identifier: GPL-2.0
/*
* linux/mm/mlock.c
*
* (C) Copyright 1995 Linus Torvalds
* (C) Copyright 2002 Christoph Hellwig
*/
/* Extracted unchanged from Linux v6.0, mm/mlock.c:213-222.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/mlock.c#L213
* License text: GPL-2.0.txt (in this directory).
*/
void mlock_page_drain_local(void)
{
struct pagevec *pvec;
local_lock(&mlock_pvec.lock);
pvec = this_cpu_ptr(&mlock_pvec.vec);
if (pagevec_count(pvec))
mlock_pagevec(pvec);
local_unlock(&mlock_pvec.lock);
}
v6.0 mlock_page_drain_remote
v6_0_mlock_page_drain_remote.c
// SPDX-License-Identifier: GPL-2.0
/*
* linux/mm/mlock.c
*
* (C) Copyright 1995 Linus Torvalds
* (C) Copyright 2002 Christoph Hellwig
*/
/* Extracted unchanged from Linux v6.0, mm/mlock.c:224-232.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/mlock.c#L224
* License text: GPL-2.0.txt (in this directory).
*/
void mlock_page_drain_remote(int cpu)
{
struct pagevec *pvec;
WARN_ON_ONCE(cpu_online(cpu));
pvec = &per_cpu(mlock_pvec.vec, cpu);
if (pagevec_count(pvec))
mlock_pagevec(pvec);
}
v6.0 lru_cache_enable
/* SPDX-License-Identifier: GPL-2.0 */
/* Extracted unchanged from Linux v6.0, include/linux/swap.h:393-396.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/include/linux/swap.h#L393
* License text: GPL-2.0.txt (in this directory).
*/
static inline void lru_cache_enable(void)
{
atomic_dec(&lru_disable_count);
}
v6.0 lru_cache_disabled
/* SPDX-License-Identifier: GPL-2.0 */
/* Extracted unchanged from Linux v6.0, include/linux/swap.h:388-391.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/include/linux/swap.h#L388
* License text: GPL-2.0.txt (in this directory).
*/
static inline bool lru_cache_disabled(void)
{
return atomic_read(&lru_disable_count);
}
참고 자료
- Lorenzo Stoakes, The Linux Memory Manager, Early Access 2025, Chapter 11, Folio Batch Drain.
- Linux v6.18 mm/swap.c: CPU별 drain과 LRU cache disable/enable.
- Linux v6.18 mm/page_alloc.c: PCP drain.
- Linux v6.18 mm/mlock.c: local·remote mlock batch drain.