dm-raid

Device Mapper와 MD를 잇는 raid target

Device-mapper RAID, 즉 `dm-raid` target은 DM에서 MD로 이어지는 bridge를 제공합니다. 이를 통해 device-mapper interface로 MD RAID driver에 접근할 수 있습니다.

Mapping target 이름은 `raid`이며 table 형식은 RAID 종류, RAID parameter와 metadata/data device 쌍을 차례로 받습니다.

  <raid_type> <#raid_params> <raid_params> \
    <#raid_devs> <metadata_dev0> <dev0> [.. <metadata_devN> <devN>]

지원 RAID type과 parity layout

  ============= ===============================================================
  raid0		RAID0 striping (no resilience)
  raid1		RAID1 mirroring
  raid4		RAID4 with dedicated last parity disk
  raid5_n 	RAID5 with dedicated last parity disk supporting takeover
		Same as raid4

		- Transitory layout
  raid5_la	RAID5 left asymmetric

		- rotating parity 0 with data continuation
  raid5_ra	RAID5 right asymmetric

		- rotating parity N with data continuation
  raid5_ls	RAID5 left symmetric

		- rotating parity 0 with data restart
  raid5_rs 	RAID5 right symmetric

		- rotating parity N with data restart
  raid6_zr	RAID6 zero restart

		- rotating parity zero (left-to-right) with data restart
  raid6_nr	RAID6 N restart

		- rotating parity N (right-to-left) with data restart
  raid6_nc	RAID6 N continue

		- rotating parity N (right-to-left) with data continuation
  raid6_n_6	RAID6 with dedicate parity disks

		- parity and Q-syndrome on the last 2 disks;
		  layout for takeover from/to raid4/raid5_n
  raid6_la_6	Same as "raid_la" plus dedicated last Q-syndrome disk

		- layout for takeover from raid5_la from/to raid6
  raid6_ra_6	Same as "raid5_ra" dedicated last Q-syndrome disk

		- layout for takeover from raid5_ra from/to raid6
  raid6_ls_6	Same as "raid5_ls" dedicated last Q-syndrome disk

		- layout for takeover from raid5_ls from/to raid6
  raid6_rs_6	Same as "raid5_rs" dedicated last Q-syndrome disk

		- layout for takeover from raid5_rs from/to raid6
  raid10        Various RAID10 inspired algorithms chosen by additional params
		(see raid10_format and raid10_copies below)

		- RAID10: Striped Mirrors (aka 'Striping on top of mirrors')
		- RAID1E: Integrated Adjacent Stripe Mirroring
		- RAID1E: Integrated Offset Stripe Mirroring
		- and other similar RAID10 variants
  ============= ===============================================================

Parity layout의 기준은 SNIA DDF Technical Position v2.0 4장입니다: `https://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf`.

필수 chunk_size와 일반 optional parameter

`<#raid_params>`는 뒤따르는 parameter 수입니다. 유일한 필수 parameter인 `<chunk_size>`가 항상 먼저 오며 sector 단위 chunk 크기, 흔히 stripe size라 부르는 값을 지정합니다.

    Mandatory parameters:
        <chunk_size>:
		      Chunk size in sectors.  This parameter is often known as
		      "stripe size".  It is the only mandatory parameter and
		      is placed first.

    followed by optional parameters (in any order):
	[sync|nosync]
		Force or prevent RAID initialization.

	[rebuild <idx>]
		Rebuild drive number 'idx' (first drive is 0).

	[daemon_sleep <ms>]
		Interval between runs of the bitmap daemon that
		clear bits.  A longer interval means less bitmap I/O but
		resyncing after a failure is likely to take longer.

	[min_recovery_rate <kB/sec/disk>]
		Throttle RAID initialization
	[max_recovery_rate <kB/sec/disk>]
		Throttle RAID initialization
	[write_mostly <idx>]
		Mark drive index 'idx' write-mostly.
	[max_write_behind <sectors>]
		See '--write-behind=' (man mdadm)
	[stripe_cache <sectors>]
		Stripe cache size (RAID 4/5/6 only)
	[region_size <sectors>]
		The region_size multiplied by the number of regions is the
		logical size of the array.  The bitmap records the device
		synchronisation state for each region.

RAID10 copies와 near·far·offset 배치

`raid10_copies <# copies>`와 `raid10_format <near|far|offset>`은 기본 RAID10 layout을 바꿉니다. Copy 기본값은 2이고 format 기본값은 `near`입니다.

		========	 ==========	   ==============
		2 drives         3 drives          4 drives
		========	 ==========	   ==============
		A1  A1           A1  A1  A2        A1  A1  A2  A2
		A2  A2           A2  A3  A3        A3  A3  A4  A4
		A3  A3           A4  A4  A5        A5  A5  A6  A6
		A4  A4           A5  A6  A6        A7  A7  A8  A8
		..  ..           ..  ..  ..        ..  ..  ..  ..
		========	 ==========	   ==============

`raid10_copies 2 raid10_format far`에서는 첫 copy 묶음 뒤에 반대 순서의 먼 copy 묶음을 배치합니다.

		========	     ============	  ===================
		2 drives             3 drives             4 drives
		========	     ============	  ===================
		A1  A2               A1   A2   A3         A1   A2   A3   A4
		A3  A4               A4   A5   A6         A5   A6   A7   A8
		A5  A6               A7   A8   A9         A9   A10  A11  A12
		..  ..               ..   ..   ..         ..   ..   ..   ..
		A2  A1               A3   A1   A2         A2   A1   A4   A3
		A4  A3               A6   A4   A5         A6   A5   A8   A7
		A6  A5               A9   A7   A8         A10  A9   A12  A11
		..  ..               ..   ..   ..         ..   ..   ..   ..
		========	     ============	  ===================

`raid10_copies 2 raid10_format offset`은 copy를 stripe 안에서 offset해 배치하며 RAID1E integrated offset stripe mirroring과 유사합니다.

		========       ==========         ================
		2 drives       3 drives           4 drives
		========       ==========         ================
		A1  A2         A1  A2  A3         A1  A2  A3  A4
		A2  A1         A3  A1  A2         A2  A1  A4  A3
		A3  A4         A4  A5  A6         A5  A6  A7  A8
		A4  A3         A6  A4  A5         A6  A5  A8  A7
		A5  A6         A7  A8  A9         A9  A10 A11 A12
		A6  A5         A9  A7  A8         A10 A9  A12 A11
		..  ..         ..  ..  ..         ..  ..  ..  ..
		========       ==========         ================

Device reshape, data_offset와 journal

`data_offset <sectors>`는 각 data device에서 user data가 시작하는 offset입니다. Stripe layout을 바꾸는 동안 원본 위에 덮어쓰지 않는 out-of-place reshape 공간을 제공하므로 언제든 interruption이나 crash가 발생해도 data를 잃지 않습니다.

`journal_dev <dev>`는 RAID4/5/6에 journal 장치를 추가해 component device의 비원자적 update가 만드는 write hole을 막습니다. 기본 writethrough이므로 non-journaled array보다 write가 제한되며, journal이 구성된 동안 takeover나 reshape는 할 수 없습니다.

`journal_mode <writethrough|writeback>`은 journaled RAID4/5/6의 cache mode입니다. Writeback journal은 자체 write hole이 없는 resilient 장치여야 합니다.

        [delta_disks <N>]
		The delta_disks option value (-251 < N < +251) triggers
		device removal (negative value) or device addition (positive
		value) to any reshape supporting raid levels 4/5/6 and 10.
		RAID levels 4/5/6 allow for addition of devices (metadata
		and data device tuple), raid10_near and raid10_offset only
		allow for device addition. raid10_far does not support any
		reshaping at all.
		A minimum of devices have to be kept to enforce resilience,
		which is 3 devices for raid4/5 and 4 devices for raid6.

        [data_offset <sectors>]
		This option value defines the offset into each data device
		where the data starts. This is used to provide out-of-place
		reshaping space to avoid writing over data while
		changing the layout of stripes, hence an interruption/crash
		may happen at any time without the risk of losing data.
		E.g. when adding devices to an existing raid set during
		forward reshaping, the out-of-place space will be allocated
		at the beginning of each raid device. The kernel raid4/5/6/10
		MD personalities supporting such device addition will read the data from
		the existing first stripes (those with smaller number of stripes)
		starting at data_offset to fill up a new stripe with the larger
		number of stripes, calculate the redundancy blocks (CRC/Q-syndrome)
		and write that new stripe to offset 0. Same will be applied to all
		N-1 other new stripes. This out-of-place scheme is used to change
		the RAID type (i.e. the allocation algorithm) as well, e.g.
		changing from raid5_ls to raid5_n.

	[journal_dev <dev>]
		This option adds a journal device to raid4/5/6 raid sets and
		uses it to close the 'write hole' caused by the non-atomic updates
		to the component devices which can cause data loss during recovery.
		The journal device is used as writethrough thus causing writes to
		be throttled versus non-journaled raid4/5/6 sets.
		Takeover/reshape is not possible with a raid4/5/6 journal device;
		it has to be deconfigured before requesting these.

	[journal_mode <mode>]
		This option sets the caching mode on journaled raid4/5/6 raid sets
		(see 'journal_dev <dev>' above) to 'writethrough' or 'writeback'.
		If 'writeback' is selected the journal device has to be resilient
		and must not suffer from the 'write hole' problem itself (e.g. use
		raid1 or raid10) to avoid a single point of failure.

Metadata/data device tuple과 missing drive

`<#raid_devs>`는 array 구성 device 수입니다. 각 device는 metadata device와 data device 두 entry로 이뤄집니다. Target 1.8.0까지 metadata/data entry를 최대 64개 지원했고, 1.9.0부터 MD kernel runtime이 강제하는 최대 253개를 지원합니다.

생성 시 drive가 실패했거나 없으면 해당 위치의 metadata와 data drive 모두에 `-`를 지정할 수 있습니다.

<#raid_devs>: The number of devices composing the array.
	Each device consists of two entries.  The first is the device
	containing the metadata (if any); the second is the one containing the
	data. A Maximum of 64 metadata/data device entries are supported
	up to target version 1.8.0.
	1.9.0 supports up to 253 which is enforced by the used MD kernel runtime.

	If a drive has failed or is missing at creation time, a '-' can be
	given for both the metadata and data drives for a given position.

RAID4 table 예제

첫 예는 metadata 장치 없이 data drive 4개와 마지막 parity drive 1개를 쓰는 RAID4입니다. Chunk는 2048 sector, 즉 1 MiB입니다.

둘째 예는 각 data drive에 대응하는 metadata device를 제공하고 `sync`로 초기화를 강제하며 disk당 최소 recovery rate를 20 KiB/s로 설정합니다.

  # RAID4 - 4 data drives, 1 parity (no metadata devices)
  # No metadata devices specified to hold superblock/bitmap info
  # Chunk size of 1MiB
  # (Lines separated for easy reading)

  0 1960893648 raid \
          raid4 1 2048 \
          5 - 8:17 - 8:33 - 8:49 - 8:65 - 8:81

  # RAID4 - 4 data drives, 1 parity (with metadata devices)
  # Chunk size of 1MiB, force RAID initialization,
  #       min recovery rate at 20 kiB/sec/disk

  0 1960893648 raid \
          raid4 4 2048 sync min_recovery_rate 20 \
          5 8:17 8:18 8:33 8:34 8:49 8:50 8:65 8:66 8:81 8:82

Table 출력과 status health·sync ratio

`dmsetup table`은 mapping을 만든 table을 표시합니다. Optional parameter는 원래 load 순서와 무관하게 문서 순서로 출력되며 `sync` 또는 `nosync`가 항상 먼저 옵니다. 반복 가능한 인자는 값 순으로 정렬됩니다.

`dmsetup status`는 array 상태와 health를 반환합니다. 첫 줄은 device-mapper 표준 출력이고 둘째·셋째 부분은 raid target이 만듭니다.

  1: <s> <l> raid \
  2:      <raid_type> <#devices> <health_chars> \
  3:      <sync_ratio> <sync_action> <mismatch_cnt>

예제 `0 1960893648 raid raid4 5 AAAAA 2/490221568 init 0`은 RAID4, device 5개가 모두 alive이며 초기 recovery가 490221568 중 2만큼 완료됐음을 뜻합니다.

        0 1960893648 raid raid4 5 AAAAA 2/490221568 init 0

	=============== =========================================================
	<raid_type>     Same as the <raid_type> used to create the array.
	<health_chars>  One char for each device, indicating:

			- 'A' = alive and in-sync
			- 'a' = alive but not in-sync
			- 'D' = dead/failed.
	<sync_ratio>    The ratio indicating how much of the array has undergone
			the process described by 'sync_action'.  If the
			'sync_action' is "check" or "repair", then the process
			of "resync" or "recover" can be considered complete.

sync_action, mismatch, offset와 journal 상태

`sync_ratio`는 해당 action의 처리 비율입니다. Action이 `check`나 `repair`라면 `resync` 또는 `recover`가 끝난 것으로 볼 수 있습니다. `mismatch_cnt`는 RAID1/10 mirror copy 차이 또는 RAID4/5/6 parity 오류 수이며 check 뒤에만 유효하고 healthy array는 0입니다.

`data_offset`은 component device별 user data 시작 위치이며 out-of-place reshape에 사용합니다.

	<sync_action>   One of the following possible states:

			idle
				- No synchronization action is being performed.
			frozen
				- The current action has been halted.
			resync
				- Array is undergoing its initial synchronization
				  or is resynchronizing after an unclean shutdown
				  (possibly aided by a bitmap).
			recover
				- A device in the array is being rebuilt or
				  replaced.
			check
				- A user-initiated full check of the array is
				  being performed.  All blocks are read and
				  checked for consistency.  The number of
				  discrepancies found are recorded in
				  <mismatch_cnt>.  No changes are made to the
				  array by this action.
			repair
				- The same as "check", but discrepancies are
				  corrected.
			reshape
				- The array is undergoing a reshape.
	<mismatch_cnt>  The number of discrepancies found between mirror copies
			in RAID1/10 or wrong parity values found in RAID4/5/6.
			This value is valid only after a "check" of the array
			is performed.  A healthy array has a 'mismatch_cnt' of 0.
	<data_offset>   The current data offset to the start of the user data on
			each component device of a raid set (see the respective
			raid parameter to support out-of-place reshaping).
	<journal_char>	- 'A' - active write-through journal device.
			- 'a' - active write-back journal device.
			- 'D' - dead journal device.
			- '-' - no journal device.
	=============== =========================================================

Message interface로 sync action 제어

`dm-raid` target은 `dmsetup` message interface로 현재 synchronization action을 제어합니다.

	========= ================================================
	"idle"    Halt the current sync action.
	"frozen"  Freeze the current sync action.
	"resync"  Initiate/continue a resync.
	"recover" Initiate/continue a recover process.
	"check"   Initiate a check (i.e. a "scrub") of the array.
	"repair"  Initiate a repair of the array.
	========= ================================================

RAID4/5/6 discard 안전성

Discard 구현은 vendor마다 다릅니다. 어떤 장치는 discarded block read에 0을 반환하고 `discard_zeroes_data`를 설정하지만, random data를 반환하거나 이 속성을 광고하고도 0을 신뢰성 있게 반환하지 않는 장치가 있습니다.

RAID4/5/6은 여러 장치 block으로 parity를 계산하며 성능상 `discard_zeroes_data`의 신뢰성에 의존합니다. Stripe 중간 block을 discard한 뒤 장치별 read 결과가 일관되지 않으면 parity가 매번 달라져 복원력에 쓸 수 없게 됩니다.

안전한 장치는 kernel whitelist에 추가되어 신뢰할 수 있습니다. 그러한 장치에서는 `devices_handle_discards_safely` dm-raid module parameter로 RAID4/5/6 discard를 안전하게 활성화할 수 있습니다.

    'devices_handle_discards_safely'

Target version 1.0.0~1.15.1 변경 이력

 1.0.0	Initial version.  Support for RAID 4/5/6
 1.1.0	Added support for RAID 1
 1.2.0	Handle creation of arrays that contain failed devices.
 1.3.0	Added support for RAID 10
 1.3.1	Allow device replacement/rebuild for RAID 10
 1.3.2	Fix/improve redundancy checking for RAID10
 1.4.0	Non-functional change.  Removes arg from mapping function.
 1.4.1	RAID10 fix redundancy validation checks (commit 55ebbb5).
 1.4.2	Add RAID10 "far" and "offset" algorithm support.
 1.5.0	Add message interface to allow manipulation of the sync_action.
	New status (STATUSTYPE_INFO) fields: sync_action and mismatch_cnt.
 1.5.1	Add ability to restore transiently failed devices on resume.
 1.5.2	'mismatch_cnt' is zero unless [last_]sync_action is "check".
 1.6.0	Add discard support (and devices_handle_discard_safely module param).
 1.7.0	Add support for MD RAID0 mappings.
 1.8.0	Explicitly check for compatible flags in the superblock metadata
	and reject to start the raid set if any are set by a newer
	target version, thus avoiding data corruption on a raid set
	with a reshape in progress.
 1.9.0	Add support for RAID level takeover/reshape/region size
	and set size reduction.
 1.9.1	Fix activation of existing RAID 4/10 mapped devices
 1.9.2	Don't emit '- -' on the status table line in case the constructor
	fails reading a superblock. Correctly emit 'maj:min1 maj:min2' and
	'D' on the status line.  If '- -' is passed into the constructor, emit
	'- -' on the table line and '-' as the status line health character.
 1.10.0	Add support for raid4/5/6 journal device
 1.10.1	Fix data corruption on reshape request
 1.11.0	Fix table line argument order
	(wrong raid10_copies/raid10_format sequence)
 1.11.1	Add raid4/5/6 journal write-back support via journal_mode option
 1.12.1	Fix for MD deadlock between mddev_suspend() and md_write_start() available
 1.13.0	Fix dev_health status at end of "recover" (was 'a', now 'A')
 1.13.1	Fix deadlock caused by early md_stop_writes().  Also fix size an
	state races.
 1.13.2	Fix raid redundancy validation and avoid keeping raid set frozen
 1.14.0	Fix reshape race on small devices.  Fix stripe adding reshape
	deadlock/potential data corruption.  Update superblock when
	specific devices are requested via rebuild.  Fix RAID leg
	rebuild errors.
 1.15.0 Fix size extensions not being synchronized in case of new MD bitmap
        pages allocated;  also fix those not occurring after previous reductions
 1.15.1 Fix argument count and arguments for rebuild/write_mostly/journal_(dev|mode)
        on the status line.