픽셀 버퍼 교환

목적과 적용 범위

이 문서는 `GPL-2.0`으로 배포되며 2021-2023 Collabora Ltd. 저작권 고지를 포함합니다.

초기 Linux graphics subsystem은 process, device, subsystem 사이에서 pixel-buffer allocation을 공유하는 기능이 매우 제한적이었습니다. 현대 system은 이 세 범주를 폭넓게 통합해야 하므로, 이 문서는 2차원 image data를 공유할 때 application과 kernel subsystem이 따라야 할 접근법을 설명합니다.

설명은 GPU와 display device의 DRM subsystem, media device의 V4L2, userspace의 Vulkan, EGL, Wayland를 기준으로 하지만 다른 subsystem도 같은 설계와 지침을 따라야 합니다.

.. SPDX-License-Identifier: GPL-2.0
.. Copyright 2021-2023 Collabora Ltd.

========================
Exchanging pixel buffers
========================

As originally designed, the Linux graphics subsystem had extremely limited
support for sharing pixel-buffer allocations between processes, devices, and
subsystems. Modern systems require extensive integration between all three
classes; this document details how applications and kernel subsystems should
approach this sharing for two-dimensional image data.

It is written with reference to the DRM subsystem for GPU and display devices,
V4L2 for media devices, and also to Vulkan, EGL and Wayland, for userspace
support, however any other subsystems should also follow this design and advice.

용어

아래 용어는 pixel buffer의 논리 image, memory storage, color 표현, memory layout을 구분하기 위한 공통 어휘입니다.

Glossary of terms
=================

.. glossary::

    image:
      Conceptually a two-dimensional array of pixels. The pixels may be stored
      in one or more memory buffers. Has width and height in pixels, pixel
      format and modifier (implicit or explicit).

    row:
      A span along a single y-axis value, e.g. from co-ordinates (0,100) to
      (200,100).

    scanline:
      Synonym for row.

    column:
      A span along a single x-axis value, e.g. from co-ordinates (100,0) to
      (100,100).

    memory buffer:
      A piece of memory for storing (parts of) pixel data. Has stride and size
      in bytes and at least one handle in some API. May contain one or more
      planes.

    plane:
      A two-dimensional array of some or all of an image's color and alpha
      channel values.

    pixel:
      A picture element. Has a single color value which is defined by one or
      more color channels values, e.g. R, G and B, or Y, Cb and Cr. May also
      have an alpha value as an additional channel.

    pixel data:
      Bytes or bits that represent some or all of the color/alpha channel values
      of a pixel or an image. The data for one pixel may be spread over several
      planes or memory buffers depending on format and modifier.

    color value:
      A tuple of numbers, representing a color. Each element in the tuple is a
      color channel value.

    color channel:
      One of the dimensions in a color model. For example, RGB model has
      channels R, G, and B. Alpha channel is sometimes counted as a color
      channel as well.

    pixel format:
      A description of how pixel data represents the pixel's color and alpha
      values.

    modifier:
      A description of how pixel data is laid out in memory buffers.

    alpha:
      A value that denotes the color coverage in a pixel. Sometimes used for
      translucency instead.

    stride:
      A value that denotes the relationship between pixel-location co-ordinates
      and byte-offset values. Typically used as the byte offset between two
      pixels at the start of vertically-consecutive tiling blocks. For linear
      layouts, the byte offset between two vertically-adjacent pixels. For
      non-linear formats the stride must be computed in a consistent way, which
      usually is done as-if the layout was linear.

    pitch:
      Synonym for stride.

Format과 modifier

모든 buffer에는 기반 format이 있어야 하며, 이 format은 각 pixel이 제공하는 color value를 설명합니다. V4L2와 fbdev처럼 subsystem마다 자체 format 설명이 있더라도 교환용 표준 설명인 `DRM_FORMAT_*` token을 가능한 한 재사용해야 합니다. 이 token은 DRM uAPI의 `drm_fourcc.h`에 정의됩니다.

각 `DRM_FORMAT_*` token은 image의 pixel coordinate와 memory buffer에 담긴 그 pixel의 color value 사이 변환을 설명합니다. RGB 또는 YUV인지, integer 또는 floating-point인지, 각 channel의 크기와 pixel memory 안 위치, color plane 사이 관계를 포함해 color channel의 수와 유형을 정의합니다.

`DRM_FORMAT_ARGB8888`에서는 pixel마다 memory에 32-bit 값 하나가 있습니다. alpha, red, green, blue channel이 각각 8-bit precision을 가지며 little-endian storage에서 most significant bit부터 least significant bit 순서로 놓입니다. `DRM_FORMAT_*`의 memory byte pattern은 CPU나 device endianness에 좌우되지 않고 format 정의 그대로이며 보통 little-endian입니다.

`DRM_FORMAT_NV12`에서는 luma와 chroma YUV sample이 별도 plane에 저장됩니다. chroma plane은 두 차원 모두 절반 resolution이므로 2x2 pixel group마다 U/V chroma sample 하나가 저장됩니다.

Format modifier는 pixel별 memory sample을 buffer의 실제 memory storage로 옮기는 변환 방식을 설명합니다. 가장 단순한 `DRM_FORMAT_MOD_LINEAR`에서는 각 plane을 왼쪽 위에서 오른쪽 아래까지 row 순서로 배치합니다. 이는 기본 교환 format이며 CPU access에 가장 편리합니다.

현대 hardware는 대개 tiled access와 경우에 따라 compression까지 사용하는 복잡한 방식을 씁니다. `DRM_FORMAT_MOD_VIVANTE_TILED`는 pixel을 row-major 순서의 4x4 block에 저장합니다. 첫 tile은 (0,0)-(3,3), 두 번째 tile은 (4,0)-(7,3)의 pixel을 포함합니다.

일부 modifier는 image에 필요한 plane 수까지 바꿉니다. 예를 들어 `I915_FORMAT_MOD_Y_TILED_CCS`는 RGB format에 두 번째 plane을 추가해 각 tile의 상태, 특히 pixel data가 완전히 채워졌는지 또는 단일 solid color에서 확장할 수 있는지를 저장합니다.

이런 확장 layout은 vendor, device generation, configuration에 매우 종속적입니다. 호환되면서 최적인 pipeline을 보장하려면 모든 사용자가 지원 modifier를 명시적으로 enumerate하고 negotiate해야 합니다.

Formats and modifiers
=====================

Each buffer must have an underlying format. This format describes the color
values provided for each pixel. Although each subsystem has its own format
descriptions (e.g. V4L2 and fbdev), the ``DRM_FORMAT_*`` tokens should be reused
wherever possible, as they are the standard descriptions used for interchange.
These tokens are described in the ``drm_fourcc.h`` file, which is a part of
DRM's uAPI.

Each ``DRM_FORMAT_*`` token describes the translation between a pixel
co-ordinate in an image, and the color values for that pixel contained within
its memory buffers. The number and type of color channels are described:
whether they are RGB or YUV, integer or floating-point, the size of each channel
and their locations within the pixel memory, and the relationship between color
planes.

For example, ``DRM_FORMAT_ARGB8888`` describes a format in which each pixel has
a single 32-bit value in memory. Alpha, red, green, and blue, color channels are
available at 8-bit precision per channel, ordered respectively from most to
least significant bits in little-endian storage. ``DRM_FORMAT_*`` is not
affected by either CPU or device endianness; the byte pattern in memory is
always as described in the format definition, which is usually little-endian.

As a more complex example, ``DRM_FORMAT_NV12`` describes a format in which luma
and chroma YUV samples are stored in separate planes, where the chroma plane is
stored at half the resolution in both dimensions (i.e. one U/V chroma
sample is stored for each 2x2 pixel grouping).

Format modifiers describe a translation mechanism between these per-pixel memory
samples, and the actual memory storage for the buffer. The most straightforward
modifier is ``DRM_FORMAT_MOD_LINEAR``, describing a scheme in which each plane
is laid out row-sequentially, from the top-left to the bottom-right corner.
This is considered the baseline interchange format, and most convenient for CPU
access.

Modern hardware employs much more sophisticated access mechanisms, typically
making use of tiled access and possibly also compression. For example, the
``DRM_FORMAT_MOD_VIVANTE_TILED`` modifier describes memory storage where pixels
are stored in 4x4 blocks arranged in row-major ordering, i.e. the first tile in
a plane stores pixels (0,0) to (3,3) inclusive, and the second tile in a plane
stores pixels (4,0) to (7,3) inclusive.

Some modifiers may modify the number of planes required for an image; for
example, the ``I915_FORMAT_MOD_Y_TILED_CCS`` modifier adds a second plane to RGB
formats in which it stores data about the status of every tile, notably
including whether the tile is fully populated with pixel data, or can be
expanded from a single solid color.

These extended layouts are highly vendor-specific, and even specific to
particular generations or configurations of devices per-vendor. For this reason,
support of modifiers must be explicitly enumerated and negotiated by all users
in order to ensure a compatible and optimal pipeline, as discussed below.

Dimension과 size

모든 pixel buffer에는 logical pixel dimension이 따라야 합니다. 이는 underlying memory storage에서 꺼내거나 저장할 수 있는 고유 sample 수를 뜻합니다. 1920x1080 `DRM_FORMAT_NV12` buffer의 luma plane에는 Y sample 1920x1080개, U/V chroma plane에는 960x540개가 있어도 전체 buffer dimension은 1920x1080입니다.

Buffer의 memory storage가 underlying memory의 base address에서 바로 시작하거나 두 dimension에 꼭 맞게 잘려 있다고 보장할 수 없습니다.

따라서 각 plane에는 byte 단위 `offset`이 필요하며, pixel별 계산 전에 memory storage base address에 더합니다. 이를 이용하면 여러 plane을 하나의 memory buffer에 합칠 수 있습니다. `DRM_FORMAT_NV12`의 luma plane은 offset 0에서 시작하고 chroma plane은 같은 buffer 안에서 해당 plane의 byte offset부터 시작할 수 있습니다.

각 plane에는 연속 row 사이의 memory offset을 나타내는 byte 단위 `stride`도 필요합니다. 1000x1000 `DRM_FORMAT_MOD_LINEAR` buffer를 alignment 때문에 1024x1000처럼 allocate했다면 logical width는 여전히 1000이고 stride는 `1024 * bpp`입니다. x-axis 양의 끝에 있는 24개 pixel 값은 유효하지 않습니다.

y dimension도 필요한 영역보다 크게 allocate하여 padding할 수 있습니다. 많은 media decoder는 height 1080 buffer를 직접 출력하지 못하고 effective height 1088이 필요합니다. 이 경우 logical height는 1080으로 유지하되 extra padding을 포함하도록 각 memory allocation을 늘립니다.

Dimensions and size
===================

Each pixel buffer must be accompanied by logical pixel dimensions. This refers
to the number of unique samples which can be extracted from, or stored to, the
underlying memory storage. For example, even though a 1920x1080
``DRM_FORMAT_NV12`` buffer has a luma plane containing 1920x1080 samples for the Y
component, and 960x540 samples for the U and V components, the overall buffer is
still described as having dimensions of 1920x1080.

The in-memory storage of a buffer is not guaranteed to begin immediately at the
base address of the underlying memory, nor is it guaranteed that the memory
storage is tightly clipped to either dimension.

Each plane must therefore be described with an ``offset`` in bytes, which will be
added to the base address of the memory storage before performing any per-pixel
calculations. This may be used to combine multiple planes into a single memory
buffer; for example, ``DRM_FORMAT_NV12`` may be stored in a single memory buffer
where the luma plane's storage begins immediately at the start of the buffer
with an offset of 0, and the chroma plane's storage follows within the same buffer
beginning from the byte offset for that plane.

Each plane must also have a ``stride`` in bytes, expressing the offset in memory
between two contiguous row. For example, a ``DRM_FORMAT_MOD_LINEAR`` buffer
with dimensions of 1000x1000 may have been allocated as if it were 1024x1000, in
order to allow for aligned access patterns. In this case, the buffer will still
be described with a width of 1000, however the stride will be ``1024 * bpp``,
indicating that there are 24 pixels at the positive extreme of the x axis whose
values are not significant.

Buffers may also be padded further in the y dimension, simply by allocating a
larger area than would ordinarily be required. For example, many media decoders
are not able to natively output buffers of height 1080, but instead require an
effective height of 1088 pixels. In this case, the buffer continues to be
described as having a height of 1080, with the memory allocation for each buffer
being increased to account for the extra padding.

지원 조합 열거

Pixel buffer를 사용하는 모든 주체는 함께 지원하는 format과 modifier 집합을 enumerate할 수 있어야 합니다. KMS에서는 DRM plane마다 `IN_FORMATS` property가 지원 DRM format과 각 format의 modifier를 나열합니다.

Userspace에서는 EGL의 `EGL_EXT_image_dma_buf_import_modifiers` extension entrypoint, Vulkan의 `VK_EXT_image_drm_format_modifier` extension, Wayland의 `zwp_linux_dmabuf_v1` extension이 이 기능을 제공합니다. 각 interface는 지원하는 format+modifier 조합 집합을 query하게 합니다.

Enumeration
===========

Every user of pixel buffers must be able to enumerate a set of supported formats
and modifiers, described together. Within KMS, this is achieved with the
``IN_FORMATS`` property on each DRM plane, listing the supported DRM formats, and
the modifiers supported for each format. In userspace, this is supported through
the `EGL_EXT_image_dma_buf_import_modifiers`_ extension entrypoints for EGL, the
`VK_EXT_image_drm_format_modifier`_ extension for Vulkan, and the
`zwp_linux_dmabuf_v1`_ extension for Wayland.

Each of these interfaces allows users to query a set of supported
format+modifier combinations.

조합 협상

사용 목적에 맞는 format+modifier 조합을 negotiate하는 책임은 userspace에 있습니다. 협상은 지원 list의 단순 intersection으로 수행합니다.

Vulkan으로 image를 render하고 KMS plane에 display하려면 먼저 해당 plane의 KMS `IN_FORMATS` property를 query합니다. 이어 physical device가 지원하는 Vulkan format을 query하되 의도한 rendering use에 해당하는 `VkImageUsageFlagBits`와 `VkImageCreateFlagBits`를 전달합니다.

두 format list를 교차해 적절한 format을 결정하고, 그 format에 대해 KMS와 Vulkan이 지원하는 modifier list를 다시 교차하면 최종 acceptable modifier list를 얻습니다.

모든 usage가 이 intersection에 참여해야 합니다. 같은 image를 video stream으로 encode하려면 사용할 media API가 지원하는 modifier 집합도 query하여 추가로 교차해야 합니다.

모든 list의 intersection이 비면 이 방식으로 buffer를 공유할 수 없습니다. 이때는 CPU access routine으로 서로 다른 usage 사이 data를 copy하는 등 성능 비용이 있는 대안을 고려해야 합니다. 결과 modifier list는 정렬되지 않으며 순서에는 의미가 없습니다.

Negotiation
===========

It is the responsibility of userspace to negotiate an acceptable format+modifier
combination for its usage. This is performed through a simple intersection of
lists. For example, if a user wants to use Vulkan to render an image to be
displayed on a KMS plane, it must:

 - query KMS for the ``IN_FORMATS`` property for the given plane
 - query Vulkan for the supported formats for its physical device, making sure
   to pass the ``VkImageUsageFlagBits`` and ``VkImageCreateFlagBits``
   corresponding to the intended rendering use
 - intersect these formats to determine the most appropriate one
 - for this format, intersect the lists of supported modifiers for both KMS and
   Vulkan, to obtain a final list of acceptable modifiers for that format

This intersection must be performed for all usages. For example, if the user
also wishes to encode the image to a video stream, it must query the media API
it intends to use for encoding for the set of modifiers it supports, and
additionally intersect against this list.

If the intersection of all lists is an empty list, it is not possible to share
buffers in this way, and an alternate strategy must be considered (e.g. using
CPU access routines to copy data between the different uses, with the
corresponding performance cost).

The resulting modifier list is unsorted; the order is not significant.

Buffer 할당

Userspace가 적절한 format과 acceptable modifier list를 결정하면 buffer를 allocate해야 합니다. Kernel이나 userspace 어디에도 universal buffer-allocation interface가 없으므로 client는 Vulkan, GBM, media API 같은 allocation interface를 임의로 선택합니다.

각 allocation request에는 최소한 pixel format, acceptable modifier list, buffer width와 height가 들어가야 합니다. API에 따라 2차원보다 많은 dimension이나 intended usage pattern 같은 property를 추가할 수 있습니다.

Allocator는 acceptable list 안에서 요청에 가장 좋다고 판단한 modifier, 필요한 padding, system memory인지 device-specific memory인지, physically contiguous한지, cache mode가 무엇인지 같은 underlying memory property를 선택합니다. 이런 memory property는 userspace에 보이지 않지만 `dma-heaps` API는 이를 다루려는 시도입니다.

Allocation 뒤 client는 allocator를 query하여 실제로 선택된 modifier와 plane별 offset 및 stride를 확인해야 합니다. Allocator는 사용 format을 바꾸거나 acceptable list 밖 modifier를 선택할 수 없고, offset, stride, size로 표현되는 padding 이외의 방식으로 pixel dimension을 바꿀 수 없습니다.

Stride나 offset alignment, 특정 memory area 안 배치 같은 추가 constraint 전달은 dma-buf의 범위 밖이며 format과 modifier token만으로 해결되지 않습니다.

Allocation
==========

Once userspace has determined an appropriate format, and corresponding list of
acceptable modifiers, it must allocate the buffer. As there is no universal
buffer-allocation interface available at either kernel or userspace level, the
client makes an arbitrary choice of allocation interface such as Vulkan, GBM, or
a media API.

Each allocation request must take, at a minimum: the pixel format, a list of
acceptable modifiers, and the buffer's width and height. Each API may extend
this set of properties in different ways, such as allowing allocation in more
than two dimensions, intended usage patterns, etc.

The component which allocates the buffer will make an arbitrary choice of what
it considers the 'best' modifier within the acceptable list for the requested
allocation, any padding required, and further properties of the underlying
memory buffers such as whether they are stored in system or device-specific
memory, whether or not they are physically contiguous, and their cache mode.
These properties of the memory buffer are not visible to userspace, however the
``dma-heaps`` API is an effort to address this.

After allocation, the client must query the allocator to determine the actual
modifier selected for the buffer, as well as the per-plane offset and stride.
Allocators are not permitted to vary the format in use, to select a modifier not
provided within the acceptable list, nor to vary the pixel dimensions other than
the padding expressed through offset, stride, and size.

Communicating additional constraints, such as alignment of stride or offset,
placement within a particular memory area, etc, is out of scope of dma-buf,
and is not solved by format and modifier tokens.

다른 영역으로 가져오기

Buffer를 다른 context, device, subsystem에서 사용하려면 format, modifier, width, height, plane별 offset과 stride를 importing API에 전달합니다.

각 memory buffer는 buffer handle로 참조하며, 한 image 안에서 handle이 고유할 수도 중복될 수도 있습니다. `DRM_FORMAT_NV12`의 luma와 chroma를 plane별 offset으로 하나의 memory buffer에 합칠 수도 있고 완전히 별도 allocation으로 둘 수도 있으므로, 모든 import와 allocation API는 plane마다 별도 handle을 제공해야 합니다.

Kernel subsystem마다 buffer management type과 interface가 다릅니다. DRM은 GEM buffer object(BO), V4L2는 자체 reference를 사용하며, 이런 native type은 context, process, device, subsystem 사이에서 이식할 수 없습니다.

이를 해결하기 위해 `dma-buf` handle을 universal buffer interchange로 사용합니다. Subsystem-specific operation으로 native buffer handle을 `dma-buf` file descriptor로 export하고 그 descriptor를 native buffer handle로 import합니다. dma-buf file descriptor는 context, process, device, subsystem 사이에 전달할 수 있습니다.

예를 들어 Wayland media player가 V4L2로 video frame을 `DRM_FORMAT_NV12` buffer에 decode하면 luma와 chroma의 두 memory plane이 userspace로 dequeue됩니다. 각 plane을 dma-buf file descriptor 하나로 export하고 format, modifier, width, height, plane별 offset/stride metadata와 함께 Wayland server로 보냅니다.

Wayland server는 같은 metadata를 사용해 descriptor를 EGL/OpenGL (ES)의 EGLImage, Vulkan의 VkImage 또는 KMS framebuffer object로 import하고 각 native buffer handle로 변환합니다. 다만 supported modifier intersection이 비어 있지 않아도 consumer가 modifier 밖의 추가 constraint를 가질 수 있으므로 모든 import 성공이 보장되지는 않습니다.

Import
======

To use a buffer within a different context, device, or subsystem, the user
passes these parameters (format, modifier, width, height, and per-plane offset
and stride) to an importing API.

Each memory buffer is referred to by a buffer handle, which may be unique or
duplicated within an image. For example, a ``DRM_FORMAT_NV12`` buffer may have
the luma and chroma buffers combined into a single memory buffer by use of the
per-plane offset parameters, or they may be completely separate allocations in
memory. For this reason, each import and allocation API must provide a separate
handle for each plane.

Each kernel subsystem has its own types and interfaces for buffer management.
DRM uses GEM buffer objects (BOs), V4L2 has its own references, etc. These types
are not portable between contexts, processes, devices, or subsystems.

To address this, ``dma-buf`` handles are used as the universal interchange for
buffers. Subsystem-specific operations are used to export native buffer handles
to a ``dma-buf`` file descriptor, and to import those file descriptors into a
native buffer handle. dma-buf file descriptors can be transferred between
contexts, processes, devices, and subsystems.

For example, a Wayland media player may use V4L2 to decode a video frame into a
``DRM_FORMAT_NV12`` buffer. This will result in two memory planes (luma and
chroma) being dequeued by the user from V4L2. These planes are then exported to
one dma-buf file descriptor per plane, these descriptors are then sent along
with the metadata (format, modifier, width, height, per-plane offset and stride)
to the Wayland server. The Wayland server will then import these file
descriptors as an EGLImage for use through EGL/OpenGL (ES), a VkImage for use
through Vulkan, or a KMS framebuffer object; each of these import operations
will take the same metadata and convert the dma-buf file descriptors into their
native buffer handles.

Having a non-empty intersection of supported modifiers does not guarantee that
import will succeed into all consumers; they may have constraints beyond those
implied by modifiers which must be satisfied.

Implicit modifier와 호환성

Modifier 개념은 DRM, V4L2 등의 기존 subsystem보다 나중에 생겼습니다. 따라서 API에 modifier support가 뒤늦게 추가되었고, 아직 modifier를 지원하지 않는 driver와 userspace를 위한 backward compatibility가 필요합니다.

EGL rendering과 KMS display 사이에 공유할 buffer를 allocate하는 GBM에는 두 entrypoint가 있습니다. `gbm_bo_create`는 format, width, height, usage token만 받고, `gbm_bo_create_with_modifiers`는 여기에 modifier list를 추가합니다.

후자는 acceptable modifier list에서 구현이 선택하거나 constraint를 만족할 수 없으면 실패합니다. 전자는 modifier가 없으므로 GBM implementation이 가장 좋을 듯한 layout을 자체적으로 고릅니다. 일부 구현은 heuristic에 따라 CPU-accessible하지 않은 tiled layout을 내부에서 선택할 수도 있으며, 선택이 적절하도록 보장할 책임은 implementation에 있습니다.

Modifier를 인식하지 않아 layout을 알 수 없는 경우를 위해 `DRM_FORMAT_MOD_INVALID` pseudo-modifier가 정의되었습니다. 이는 layout이 알려지지 않았고 underlying layout 판단에 driver 자체 logic을 사용해야 한다는 선언입니다.

주의할 점은 `DRM_FORMAT_MOD_INVALID`가 0이 아니라는 것입니다. Modifier 0은 image가 linear layout이라는 명시적 보장인 `DRM_FORMAT_MOD_LINEAR`입니다. Default value 0을 no modifier 또는 linear modifier와 혼동하면 안 됩니다. `DRM_IOCTL_MODE_ADDFB2`처럼 일부 API에서는 invalid modifier를 out-of-band flag로 표시합니다.

Enumeration에서는 interface가 `DRM_FORMAT_MOD_INVALID`만 반환해 explicit modifier 미지원임을 알리거나, 더 큰 list에 포함해 implicit modifier도 사용할 수 있음을 알릴 수 있습니다.

Allocation에서는 user가 `DRM_FORMAT_MOD_INVALID`만 제공하여 modifier list를 전혀 주지 않은 것과 같이 explicit modifier를 금지할 수 있고, 더 큰 list에 포함해 implicit modifier allocation도 acceptable하다고 선언할 수 있습니다.

Post-allocation query에서는 underlying layout이 implementation-defined이고 explicit description이 없음을 나타내기 위해 allocator가 `DRM_FORMAT_MOD_INVALID`를 반환할 수 있습니다. 이는 user가 acceptable list에 INVALID를 넣었거나 list를 주지 않았을 때만 허용됩니다.

Import에서는 modifier를 알 수 없다는 뜻으로 `DRM_FORMAT_MOD_INVALID`를 전달하거나 modifier를 생략할 수 있습니다. 이는 buffer가 explicit modifier로 allocate되지 않았을 때만 acceptable합니다.

따라서 하나의 buffer에서 producer와 모든 consumer가 이루는 전체 operation chain은 완전히 implicit하거나 완전히 explicit해야 합니다. GPU, display, media가 함께 쓸 buffer에서 media API가 modifier를 지원하지 않는다면 explicit modifier로 allocate한 뒤 media API에는 modifier 없이 import해서는 안 됩니다. 전체를 implicit으로 allocate하거나 media용 buffer를 별도로 allocate하고 두 buffer 사이를 copy해야 합니다.

예외적으로 implicit allocation을 explicit modifier로 upgrade할 수 있습니다. `gbm_bo_create`로 modifier 없이 allocate한 뒤 `gbm_bo_get_modifier`로 modifier를 query하여 valid modifier가 반환되면 이를 explicit modifier token으로 사용할 수 있습니다.

서로 다른 사용자 사이 교환용 buffer를 modifier 없이 allocate할 때는 universal baseline인 `DRM_FORMAT_MOD_LINEAR` 사용을 강하게 권장합니다. 다만 implicit modifier operation에는 driver-specific heuristic이 적용될 수 있어 buffer content가 올바르게 해석된다는 보장은 없습니다.

Buffer를 교환하려는 새 userspace program, protocol, kernel subsystem은 memory plane용 dma-buf file descriptor, format을 나타내는 DRM format token, memory layout을 나타내는 DRM format modifier, 최소 width와 height, plane마다 최소 offset과 stride를 제공하여 interoperability를 보장해야 합니다.

Implicit modifiers
==================

The concept of modifiers post-dates all of the subsystems mentioned above. As
such, it has been retrofitted into all of these APIs, and in order to ensure
backwards compatibility, support is needed for drivers and userspace which do
not (yet) support modifiers.

As an example, GBM is used to allocate buffers to be shared between EGL for
rendering and KMS for display. It has two entrypoints for allocating buffers:
``gbm_bo_create`` which only takes the format, width, height, and a usage token,
and ``gbm_bo_create_with_modifiers`` which extends this with a list of modifiers.

In the latter case, the allocation is as discussed above, being provided with a
list of acceptable modifiers that the implementation can choose from (or fail if
it is not possible to allocate within those constraints). In the former case
where modifiers are not provided, the GBM implementation must make its own
choice as to what is likely to be the 'best' layout. Such a choice is entirely
implementation-specific: some will internally use tiled layouts which are not
CPU-accessible if the implementation decides that is a good idea through
whatever heuristic. It is the implementation's responsibility to ensure that
this choice is appropriate.

To support this case where the layout is not known because there is no awareness
of modifiers, a special ``DRM_FORMAT_MOD_INVALID`` token has been defined. This
pseudo-modifier declares that the layout is not known, and that the driver
should use its own logic to determine what the underlying layout may be.

.. note::

  ``DRM_FORMAT_MOD_INVALID`` is a non-zero value. The modifier value zero is
  ``DRM_FORMAT_MOD_LINEAR``, which is an explicit guarantee that the image
  has the linear layout. Care and attention should be taken to ensure that
  zero as a default value is not mixed up with either no modifier or the linear
  modifier. Also note that in some APIs the invalid modifier value is specified
  with an out-of-band flag, like in ``DRM_IOCTL_MODE_ADDFB2``.

There are four cases where this token may be used:
  - during enumeration, an interface may return ``DRM_FORMAT_MOD_INVALID``, either
    as the sole member of a modifier list to declare that explicit modifiers are
    not supported, or as part of a larger list to declare that implicit modifiers
    may be used
  - during allocation, a user may supply ``DRM_FORMAT_MOD_INVALID``, either as the
    sole member of a modifier list (equivalent to not supplying a modifier list
    at all) to declare that explicit modifiers are not supported and must not be
    used, or as part of a larger list to declare that an allocation using implicit
    modifiers is acceptable
  - in a post-allocation query, an implementation may return
    ``DRM_FORMAT_MOD_INVALID`` as the modifier of the allocated buffer to declare
    that the underlying layout is implementation-defined and that an explicit
    modifier description is not available; per the above rules, this may only be
    returned when the user has included ``DRM_FORMAT_MOD_INVALID`` as part of the
    list of acceptable modifiers, or not provided a list
  - when importing a buffer, the user may supply ``DRM_FORMAT_MOD_INVALID`` as the
    buffer modifier (or not supply a modifier) to indicate that the modifier is
    unknown for whatever reason; this is only acceptable when the buffer has
    not been allocated with an explicit modifier

It follows from this that for any single buffer, the complete chain of operations
formed by the producer and all the consumers must be either fully implicit or fully
explicit. For example, if a user wishes to allocate a buffer for use between
GPU, display, and media, but the media API does not support modifiers, then the
user **must not** allocate the buffer with explicit modifiers and attempt to
import the buffer into the media API with no modifier, but either perform the
allocation using implicit modifiers, or allocate the buffer for media use
separately and copy between the two buffers.

As one exception to the above, allocations may be 'upgraded' from implicit
to explicit modifiers. For example, if the buffer is allocated with
``gbm_bo_create`` (taking no modifiers), the user may then query the modifier with
``gbm_bo_get_modifier`` and then use this modifier as an explicit modifier token
if a valid modifier is returned.

When allocating buffers for exchange between different users and modifiers are
not available, implementations are strongly encouraged to use
``DRM_FORMAT_MOD_LINEAR`` for their allocation, as this is the universal baseline
for exchange. However, it is not guaranteed that this will result in the correct
interpretation of buffer content, as implicit modifier operation may still be
subject to driver-specific heuristics.

Any new users - userspace programs and protocols, kernel subsystems, etc -
wishing to exchange buffers must offer interoperability through dma-buf file
descriptors for memory planes, DRM format tokens to describe the format, DRM
format modifiers to describe the layout in memory, at least width and height for
dimensions, and at least offset and stride for each memory plane.

.. _zwp_linux_dmabuf_v1: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml
.. _VK_EXT_image_drm_format_modifier: https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_EXT_image_drm_format_modifier.html
.. _EGL_EXT_image_dma_buf_import_modifiers: https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import_modifiers.txt