요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
Target interface
cache.rst:173-337Constructor, status field, policy message, cache-block invalidation과 dmsetup 예제를 설명합니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
=====
Cache
=====
Introduction
============
dm-cache is a device mapper target written by Joe Thornber, Heinz
Mauelshagen, and Mike Snitzer.
It aims to improve performance of a block device (eg, a spindle) by
dynamically migrating some of its data to a faster, smaller device
(eg, an SSD).
This device-mapper solution allows us to insert this caching at
different levels of the dm stack, for instance above the data device for
a thin-provisioning pool. Caching solutions that are integrated more
closely with the virtual memory system should give better performance.
The target reuses the metadata library used in the thin-provisioning
library.
The decision as to what data to migrate and when is left to a plug-in
policy module. Several of these have been written as we experiment,
and we hope other people will contribute others for specific io
scenarios (eg. a vm image server).
Glossary
========
Migration
Movement of the primary copy of a logical block from one
device to the other.
Promotion
Migration from slow device to fast device.
Demotion
Migration from fast device to slow device.
The origin device always contains a copy of the logical block, which
may be out of date or kept in sync with the copy on the cache device
(depending on policy).
Design
======
Sub-devices
-----------
The target is constructed by passing three devices to it (along with
other parameters detailed later):
1. An origin device - the big, slow one.
2. A cache device - the small, fast one.
3. A small metadata device - records which blocks are in the cache,
which are dirty, and extra hints for use by the policy object.
This information could be put on the cache device, but having it
separate allows the volume manager to configure it differently,
e.g. as a mirror for extra robustness. This metadata device may only
be used by a single cache device.
Fixed block size
----------------
The origin is divided up into blocks of a fixed size. This block size
is configurable when you first create the cache. Typically we've been
using block sizes of 256KB - 1024KB. The block size must be between 64
sectors (32KB) and 2097152 sectors (1GB) and a multiple of 64 sectors (32KB).
Having a fixed block size simplifies the target a lot. But it is
something of a compromise. For instance, a small part of a block may be
getting hit a lot, yet the whole block will be promoted to the cache.
So large block sizes are bad because they waste cache space. And small
block sizes are bad because they increase the amount of metadata (both
in core and on disk).
Cache operating modes
---------------------
The cache has three operating modes: writeback, writethrough and
passthrough.
If writeback, the default, is selected then a write to a block that is
cached will go only to the cache and the block will be marked dirty in
the metadata.
If writethrough is selected then a write to a cached block will not
complete until it has hit both the origin and cache devices. Clean
blocks should remain clean.
If passthrough is selected, useful when the cache contents are not known
to be coherent with the origin device, then all reads are served from
the origin device (all reads miss the cache) and all writes are
forwarded to the origin device; additionally, write hits cause cache
block invalidates. To enable passthrough mode the cache must be clean.
Passthrough mode allows a cache device to be activated without having to
worry about coherency. Coherency that exists is maintained, although
the cache will gradually cool as writes take place. If the coherency of
the cache can later be verified, or established through use of the
"invalidate_cblocks" message, the cache device can be transitioned to
writethrough or writeback mode while still warm. Otherwise, the cache
contents can be discarded prior to transitioning to the desired
operating mode.
A simple cleaner policy is provided, which will clean (write back) all
dirty blocks in a cache. Useful for decommissioning a cache or when
shrinking a cache. Shrinking the cache's fast device requires all cache
blocks, in the area of the cache being removed, to be clean. If the
area being removed from the cache still contains dirty blocks the resize
will fail. Care must be taken to never reduce the volume used for the
cache's fast device until the cache is clean. This is of particular
importance if writeback mode is used. Writethrough and passthrough
modes already maintain a clean cache. Future support to partially clean
the cache, above a specified threshold, will allow for keeping the cache
warm and in writeback mode during resize.
Migration throttling
--------------------
Migrating data between the origin and cache device uses bandwidth.
The user can set a throttle to prevent more than a certain amount of
migration occurring at any one time. Currently we're not taking any
account of normal io traffic going to the devices. More work needs
doing here to avoid migrating during those peak io moments.
For the time being, a message "migration_threshold <#sectors>"
can be used to set the maximum number of sectors being migrated,
the default being 2048 sectors (1MB).
Updating on-disk metadata
-------------------------
On-disk metadata is committed every time a FLUSH or FUA bio is written.
If no such requests are made then commits will occur every second. This
means the cache behaves like a physical disk that has a volatile write
cache. If power is lost you may lose some recent writes. The metadata
should always be consistent in spite of any crash.
The 'dirty' state for a cache block changes far too frequently for us
to keep updating it on the fly. So we treat it as a hint. In normal
operation it will be written when the dm device is suspended. If the
system crashes all cache blocks will be assumed dirty when restarted.
Per-block policy hints
----------------------
Policy plug-ins can store a chunk of data per cache block. It's up to
the policy how big this chunk is, but it should be kept small. Like the
dirty flags this data is lost if there's a crash so a safe fallback
value should always be possible.
Policy hints affect performance, not correctness.
Policy messaging
----------------
Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these. Device-mapper
messages are used. Refer to cache-policies.txt.
Discard bitset resolution
-------------------------
We can avoid copying data during migration if we know the block has
been discarded. A prime example of this is when mkfs discards the
whole block device. We store a bitset tracking the discard state of
blocks. However, we allow this bitset to have a different block size
from the cache blocks. This is because we need to track the discard
state for all of the origin device (compare with the dirty bitset
which is just for the smaller cache device).
Target interface
================
Constructor
-----------
::
cache <metadata dev> <cache dev> <origin dev> <block size>
<#feature args> [<feature arg>]*
<policy> <#policy args> [policy args]*
================ =======================================================
metadata dev fast device holding the persistent metadata
cache dev fast device holding cached data blocks
origin dev slow device holding original data blocks
block size cache unit size in sectors
#feature args number of feature arguments passed
feature args writethrough or passthrough (The default is writeback.)
policy the replacement policy to use
#policy args an even number of arguments corresponding to
key/value pairs passed to the policy
policy args key/value pairs passed to the policy
E.g. 'sequential_threshold 1024'
See cache-policies.txt for details.
================ =======================================================
Optional feature arguments are:
==================== ========================================================
writethrough write through caching that prohibits cache block
content from being different from origin block content.
Without this argument, the default behaviour is to write
back cache block contents later for performance reasons,
so they may differ from the corresponding origin blocks.
passthrough a degraded mode useful for various cache coherency
situations (e.g., rolling back snapshots of
underlying storage). Reads and writes always go to
the origin. If a write goes to a cached origin
block, then the cache block is invalidated.
To enable passthrough mode the cache must be clean.
metadata2 use version 2 of the metadata. This stores the dirty
bits in a separate btree, which improves speed of
shutting down the cache.
no_discard_passdown disable passing down discards from the cache
to the origin's data device.
==================== ========================================================
A policy called 'default' is always registered. This is an alias for
the policy we currently think is giving best all round performance.
As the default policy could vary between kernels, if you are relying on
the characteristics of a specific policy, always request it by name.
Status
------
::
<metadata block size> <#used metadata blocks>/<#total metadata blocks>
<cache block size> <#used cache blocks>/<#total cache blocks>
<#read hits> <#read misses> <#write hits> <#write misses>
<#demotions> <#promotions> <#dirty> <#features> <features>*
<#core args> <core args>* <policy name> <#policy args> <policy args>*
<cache metadata mode>
========================= =====================================================
metadata block size Fixed block size for each metadata block in
sectors
#used metadata blocks Number of metadata blocks used
#total metadata blocks Total number of metadata blocks
cache block size Configurable block size for the cache device
in sectors
#used cache blocks Number of blocks resident in the cache
#total cache blocks Total number of cache blocks
#read hits Number of times a READ bio has been mapped
to the cache
#read misses Number of times a READ bio has been mapped
to the origin
#write hits Number of times a WRITE bio has been mapped
to the cache
#write misses Number of times a WRITE bio has been
mapped to the origin
#demotions Number of times a block has been removed
from the cache
#promotions Number of times a block has been moved to
the cache
#dirty Number of blocks in the cache that differ
from the origin
#feature args Number of feature args to follow
feature args 'writethrough' (optional)
#core args Number of core arguments (must be even)
core args Key/value pairs for tuning the core
e.g. migration_threshold
policy name Name of the policy
#policy args Number of policy arguments to follow (must be even)
policy args Key/value pairs e.g. sequential_threshold
cache metadata mode ro if read-only, rw if read-write
In serious cases where even a read-only mode is
deemed unsafe no further I/O will be permitted and
the status will just contain the string 'Fail'.
The userspace recovery tools should then be used.
needs_check 'needs_check' if set, '-' if not set
A metadata operation has failed, resulting in the
needs_check flag being set in the metadata's
superblock. The metadata device must be
deactivated and checked/repaired before the
cache can be made fully operational again.
'-' indicates needs_check is not set.
========================= =====================================================
Messages
--------
Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these. Device-mapper
messages are used. (A sysfs interface would also be possible.)
The message format is::
<key> <value>
E.g.::
dmsetup message my_cache 0 sequential_threshold 1024
Invalidation is removing an entry from the cache without writing it
back. Cache blocks can be invalidated via the invalidate_cblocks
message, which takes an arbitrary number of cblock ranges. Each cblock
range's end value is "one past the end", meaning 5-10 expresses a range
of values from 5 to 9. Each cblock must be expressed as a decimal
value, in the future a variant message that takes cblock ranges
expressed in hexadecimal may be needed to better support efficient
invalidation of larger caches. The cache must be in passthrough mode
when invalidate_cblocks is used::
invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]*
E.g.::
dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789
Examples
========
The test suite can be found here:
https://github.com/jthornber/device-mapper-test-suite
::
dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
/dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'
dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
/dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \
mq 4 sequential_threshold 1024 random_threshold 8'
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
dm-cache 목적과 policy 분리
1-27`dm-cache`는 Joe Thornber, Heinz Mauelshagen, Mike Snitzer가 작성한 device-mapper target입니다. Spindle 같은 block device의 일부 data를 SSD 같은 더 작고 빠른 device로 동적으로 옮겨 성능을 높이는 것이 목적입니다.
Device-mapper 방식이므로 thin-provisioning pool의 data device 위처럼 DM stack의 여러 level에 cache를 끼워 넣을 수 있습니다. Virtual memory system과 더 밀접하게 통합된 cache solution이 더 나은 성능을 낼 수 있습니다.
Target은 thin-provisioning library가 쓰는 metadata library를 재사용합니다. 어떤 data를 언제 migration할지는 plug-in policy module에 맡깁니다. 여러 실험 policy가 작성되었고, VM image server 같은 특정 I/O scenario용 policy도 기여할 수 있습니다.
큰 origin의 일부 logical block을 작은 fast cache에 동적으로 배치합니다.
Migration, promotion, demotion
28-42Logical block의 primary copy가 어느 방향으로 이동하는지를 구분합니다.
Origin device에는 logical block의 copy가 항상 남습니다. Policy에 따라 그 copy는 cache device의 copy보다 오래된 상태일 수도 있고, cache copy와 계속 동기화될 수도 있습니다.
Origin, cache, metadata device
43-62Cache target은 뒤에서 설명할 parameter와 함께 origin device, cache device, metadata device의 세 device를 받아 구성합니다.
Data와 persistent state를 역할별 device에 나눕니다.
고정 cache block 크기의 절충
63-77Origin은 처음 cache를 만들 때 정한 고정 크기 block으로 나뉩니다. 보통 256KB에서 1024KB를 사용합니다. 허용 범위는 64 sector(32KB)에서 `2097152` sector(1GB)이고, 64 sector(32KB)의 배수여야 합니다.
고정 block 크기는 target을 크게 단순화하지만 절충이 필요합니다. Block 일부만 자주 접근되어도 전체 block을 promotion합니다. 큰 block은 cache 공간을 낭비하고, 작은 block은 memory와 disk 양쪽의 metadata 양을 늘립니다.
Workload와 metadata 비용 사이의 절충입니다.
writeback, writethrough, passthrough
78-117Cache에는 `writeback`, `writethrough`, `passthrough` 세 operating mode가 있습니다.
Write 처리와 origin/cache coherence 방식이 다릅니다.
Passthrough는 cache 내용이 origin과 coherent한지 알 수 없을 때 유용합니다. 현재 coherence는 유지되지만 write가 진행될수록 cache는 점차 식습니다. 나중에 coherence를 검증하거나 `invalidate_cblocks` message로 확립하면 warm 상태를 유지한 채 writethrough나 writeback으로 바꿀 수 있습니다. 그렇지 않으면 원하는 mode로 바꾸기 전에 cache 내용을 버려야 합니다.
단순한 `cleaner` policy는 dirty block을 모두 writeback합니다. Cache를 폐기하거나 줄일 때 유용합니다. Fast device에서 제거할 영역의 cache block이 모두 clean하지 않으면 resize가 실패하므로 cache가 clean해지기 전에는 fast volume을 절대 줄이면 안 됩니다. 특히 writeback mode에서 중요하며, writethrough와 passthrough는 이미 clean cache를 유지합니다. 향후에는 지정 threshold 위쪽만 부분적으로 clean해 resize 중에도 cache를 warm writeback 상태로 유지할 수 있도록 할 예정입니다.
제거할 fast-device 영역에 dirty block이 남지 않도록 먼저 정리합니다.
Migration throttle과 on-disk metadata
118-144Origin과 cache 사이의 data migration은 bandwidth를 씁니다. 동시에 이동할 sector 수를 제한하는 throttle을 설정할 수 있습니다. 현재는 device로 향하는 일반 I/O traffic을 고려하지 않으므로 peak I/O 시간의 migration을 피하는 개선이 필요합니다.
현재는 `migration_threshold <#sectors>` message로 동시에 migration할 최대 sector 수를 정하며 기본값은 2048 sector(1MB)입니다.
FLUSH 또는 FUA bio가 write될 때마다 on-disk metadata를 commit합니다. 그런 request가 없으면 1초마다 commit합니다. 따라서 volatile write cache가 있는 물리 disk처럼 동작하며, 전원을 잃으면 최근 write 일부를 잃을 수 있지만 crash 뒤에도 metadata 자체는 일관되어야 합니다.
Cache block의 `dirty` 상태는 너무 자주 바뀌므로 매번 즉시 disk에 갱신하지 않고 hint로 취급합니다. 정상 동작에서는 DM device를 suspend할 때 기록합니다. System이 crash하면 restart 시 모든 cache block을 dirty로 간주합니다.
정상 commit trigger와 dirty hint의 보수적 복구 방식입니다.
Policy hint, message와 discard bitset
145-172Policy plug-in은 cache block마다 작은 data chunk를 저장할 수 있습니다. 크기는 policy가 정하지만 작게 유지해야 합니다. Dirty flag처럼 crash 시 사라지므로 항상 안전한 fallback 값을 만들 수 있어야 합니다. Policy hint는 correctness가 아니라 performance에만 영향을 줍니다.
Policy마다 tunable이 다르므로 이를 읽고 설정하는 generic mechanism으로 device-mapper message를 사용합니다. 자세한 내용은 `cache-policies.txt`를 참고합니다.
Block이 discard되었음을 알면 migration 때 data copy를 생략할 수 있습니다. 대표적으로 `mkfs`가 block device 전체를 discard하는 경우가 있습니다. 이를 위해 block의 discard state를 bitset으로 추적합니다.
Discard bitset은 cache block과 다른 block 크기를 쓸 수 있습니다. Dirty bitset은 더 작은 cache device만 추적하지만 discard state는 origin device 전체를 추적해야 하기 때문입니다.
Discard state가 알려진 영역은 불필요한 data copy를 건너뜁니다.
Cache target constructor
173-232Cache target의 constructor table 형식은 다음과 같습니다.
cache <metadata dev> <cache dev> <origin dev> <block size>
<#feature args> [<feature arg>]*
<policy> <#policy args> [policy args]*Device, block size, feature와 policy를 순서대로 지정합니다.
Constructor에서 cache의 coherence와 metadata 동작을 조정합니다.
`default`라는 policy는 항상 등록되며 현재 가장 전반적인 성능이 좋다고 판단한 policy의 alias입니다. Kernel에 따라 default가 바뀔 수 있으므로 특정 policy 특성에 의존한다면 이름을 명시해야 합니다.
Status 출력과 복구 신호
233-291Cache target status는 metadata/cache 사용량, hit/miss, migration, dirty block, feature와 policy, metadata mode를 다음 순서로 출력합니다.
<metadata block size> <#used metadata blocks>/<#total metadata blocks>
<cache block size> <#used cache blocks>/<#total cache blocks>
<#read hits> <#read misses> <#write hits> <#write misses>
<#demotions> <#promotions> <#dirty> <#features> <features>*
<#core args> <core args>* <policy name> <#policy args> <policy args>*
<cache metadata mode>Status line의 각 field가 나타내는 값입니다.
Policy message, cache block 무효화와 생성 예
292-337서로 다른 policy tunable을 generic하게 읽고 설정하기 위해 device-mapper message를 사용합니다. Sysfs interface도 가능하지만 이 target은 message 형식을 사용합니다.
<key> <value>예를 들어 다음 명령으로 sequential threshold를 설정합니다.
dmsetup message my_cache 0 sequential_threshold 1024Invalidation은 cache entry를 writeback하지 않고 제거하는 동작입니다. `invalidate_cblocks` message는 임의 개수의 cache-block range를 받습니다. Range의 end는 포함하지 않는 one-past-the-end 값이므로 `5-10`은 5부터 9까지입니다. 각 cblock은 decimal로 써야 하며, 큰 cache를 효율적으로 무효화하려면 향후 hexadecimal range variant가 필요할 수 있습니다. 이 message를 사용할 때 cache는 passthrough mode여야 합니다.
invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]*아래 예는 cblock `2345`, `3456-4566`, `5678-6788`을 무효화합니다.
dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789Test suite는 `https://github.com/jthornber/device-mapper-test-suite`에 있습니다. 다음 두 constructor 예는 각각 block size 512의 default policy와 block size 1024의 mq policy를 사용합니다.
dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
/dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'
dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
/dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \
mq 4 sequential_threshold 1024 random_threshold 8'Passthrough mode에서 지정 cache block을 writeback 없이 제거합니다.
구성과 data lifecycle
cache.rst:1-172Origin·cache·metadata device, block 크기, 세 operating mode, migration과 metadata 일관성 모델을 정리합니다.