Multiplane Overlay (MPO)

MPO 목적과 DRM atomic model

Multiplane Overlay(MPO)는 여러 framebuffer를 graphics 또는 compute shader로 합성하지 않고 display controller의 fixed-function hardware에서 합성합니다. 그 결과 별도 composition shader와 buffer copy가 필요하지 않아 GPU와 CPU 부하가 줄고, graphics/compute pipeline을 저전력 상태에 둘 수 있습니다. 각 plane은 global compositor의 page-flip present rate에 묶이지 않고 독립적인 timing으로 page flip할 수 있으므로 latency도 줄일 수 있습니다.

MPO는 DRM atomic model에서만 사용할 수 있습니다. Userspace는 `drmModeAtomicCommit` 한 번으로 modesetting, page-flipping 등 display hardware configuration을 제출하고, `drmModeGetResources`로 plane, CRTC, connector의 개수와 resource 제한을 조회합니다.

원문 예시 ASIC은 CRTC마다 primary 1개와 cursor 1개를 제공해 각각 4개이며, 모든 CRTC가 공유하는 overlay 1개를 제공합니다. 실제 plane 개수는 ASIC마다 달라집니다. 예시 hardware에는 pipe 4개가 있고 일반적인 AMD single-display configuration은 plane 하나에 pipe 2개를 쓰는 pipe-split을 사용할 수 있습니다.

각 primary 또는 overlay plane은 최소 pipe 1개를 필요로 합니다. 따라서 이 가상 hardware에서는 모든 CRTC를 합쳐 최대 4개 plane만 활성화할 수 있고, 이를 넘는 atomic commit은 거부됩니다. 이 숫자는 개념 예시일 뿐이며 실제 제한은 DCN 세대와 ASIC마다 다릅니다.

========================
Multiplane Overlay (MPO)
========================

.. note:: You will get more from this page if you have already read the
   'Documentation/gpu/amdgpu/display/dcn-overview.rst'.


Multiplane Overlay (MPO) allows for multiple framebuffers to be composited via
fixed-function hardware in the display controller rather than using graphics or
compute shaders for composition. This can yield some power savings if it means
the graphics/compute pipelines can be put into low-power states. In summary,
MPO can bring the following benefits:

* Decreased GPU and CPU workload - no composition shaders needed, no extra
  buffer copy needed, GPU can remain idle.
* Plane independent page flips - No need to be tied to global compositor
  page-flip present rate, reduced latency, independent timing.

.. note:: Keep in mind that MPO is all about power-saving; if you want to learn
   more about power-save in the display context, check the link:
   `Power <https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/power.rst>`__.

Multiplane Overlay is only available using the DRM atomic model. The atomic
model only uses a single userspace IOCTL for configuring the display hardware
(modesetting, page-flipping, etc) - drmModeAtomicCommit. To query hardware
resources and limitations userspace also calls into drmModeGetResources which
reports back the number of planes, CRTCs, and connectors. There are three types
of DRM planes that the driver can register and work with:

* ``DRM_PLANE_TYPE_PRIMARY``: Primary planes represent a "main" plane for a
  CRTC, primary planes are the planes operated upon by CRTC modesetting and
  flipping operations.
* ``DRM_PLANE_TYPE_CURSOR``: Cursor planes represent a "cursor" plane for a
  CRTC. Cursor planes are the planes operated upon by the cursor IOCTLs
* ``DRM_PLANE_TYPE_OVERLAY``: Overlay planes represent all non-primary,
  non-cursor planes. Some drivers refer to these types of planes as "sprites"
  internally.

To illustrate how it works, let's take a look at a device that exposes the
following planes to userspace:

* 4 Primary planes (1 per CRTC).
* 4 Cursor planes (1 per CRTC).
* 1 Overlay plane (shared among CRTCs).

.. note:: Keep in mind that different ASICs might expose other numbers of
   planes.

For this hardware example, we have 4 pipes (if you don't know what AMD pipe
means, look at 'Documentation/gpu/amdgpu/display/dcn-overview.rst', section
"AMD Hardware Pipeline"). Typically most AMD devices operate in a pipe-split
configuration for optimal single display output (e.g., 2 pipes per plane).

A typical MPO configuration from userspace - 1 primary + 1 overlay on a single
display - will see 4 pipes in use, 2 per plane.

At least 1 pipe must be used per plane (primary and overlay), so for this
hypothetical hardware that we are using as an example, we have an absolute
limit of 4 planes across all CRTCs. Atomic commits will be rejected for display
configurations using more than 4 planes. Again, it is important to stress that
every DCN has different restrictions; here, we are just trying to provide the
concept idea.

Plane format, 위치와 scaling 제한

AMDGPU는 atomic commit 단계에서 plane format, destination rectangle과 scaling ratio를 검증합니다. 제한을 따르지 않는 commit은 부분 적용되지 않고 전체가 거부됩니다.

Plane Restrictions
==================

AMDGPU imposes restrictions on the use of DRM planes in the driver.

Atomic commits will be rejected for commits which do not follow these
restrictions:

* Overlay planes must be in ARGB8888 or XRGB8888 format
* Planes cannot be placed outside of the CRTC destination rectangle
* Planes cannot be downscaled more than 1/4x of their original size
* Planes cannot be upscaled more than 16x of their original size

Not every property is available on every plane:

* Only primary planes have color-space and non-RGB format support
* Only overlay planes have alpha blending support

AMD cursor 연결 제약

`mpo-cursor.svg`의 왼쪽은 DRM이 cursor와 Plane 1, Plane 2를 독립 object로 합성해 CRTC로 보내는 모델을 나타냅니다. 오른쪽 AMD hardware에서는 cursor가 독립 hardware plane이 아니라 다른 plane에 붙고, 그 parent plane이 CRTC로 연결됩니다.

따라서 MPO 구성에서는 legacy API로 cursor plane을 설정하지 않아야 합니다. AMD hardware에는 dedicated cursor plane이 없으며 cursor는 parent plane의 scaling과 color processing을 상속한다는 사실이 핵심입니다.

Cursor Restrictions
===================

Before we start to describe some restrictions around cursor and MPO, see the
below image:

.. kernel-figure:: mpo-cursor.svg

The image on the left side represents how DRM expects the cursor and planes to
be blended. However, AMD hardware handles cursors differently, as you can see
on the right side; basically, our cursor cannot be drawn outside its associated
plane as it is being treated as part of the plane. Another consequence of that
is that cursors inherit the color and scale from the plane.

As a result of the above behavior, do not use legacy API to set up the cursor
plane when working with MPO; otherwise, you might encounter unexpected
behavior.

In short, AMD HW has no dedicated cursor planes. A cursor is attached to
another plane and therefore inherits any scaling or color processing from its
parent plane.

PIP primary-underlay 구성과 IGT 검증

Picture-in-Picture(PIP) video playback은 `primary plane as underlay` 전략을 사용합니다. YUV primary plane이 video를 직접 scanout하고, RGBA overlay plane이 desktop UI, player control과 subtitle을 담습니다. 현재 driver는 이 기본 2-plane 구성을 지원하며 더 많은 plane으로의 확장은 아직 지원하지 않습니다.

Single-video에서는 video buffer를 primary plane에 직접 사용하고 `CRTC_X`, `CRTC_Y`, `CRTC_W`, `CRTC_H`로 desktop 안의 위치와 크기를 정합니다. Primary에는 source content에 맞춘 `COLOR_RANGE`, `COLOR_ENCODING`도 설정해야 합니다. Overlay는 CRTC native size여야 하며, desktop buffer는 정적으로 유지하면서 primary framebuffer만 standard double buffering으로 교체할 수 있습니다.

여러 video를 한 primary plane에 담을 때 compositor는 CRTC native size의 YUV buffer를 만들고 각 video buffer를 여기에 합성해 direct YUV scanout합니다. Plane 전체가 하나의 `COLOR_RANGE`와 `COLOR_ENCODING`을 사용하므로 모든 source의 color space와 encoding이 서로 맞아야 합니다. Overlay 역시 CRTC native size를 사용하고 각 video 위치에 transparent cutout을 만듭니다.

Use Cases
=========

Picture-in-Picture (PIP) playback - Underlay strategy
-----------------------------------------------------

Video playback should be done using the "primary plane as underlay" MPO
strategy. This is a 2 planes configuration:

* 1 YUV DRM Primary Plane (e.g. NV12 Video)
* 1 RGBA DRM Overlay Plane (e.g. ARGB8888 desktop). The compositor should
  prepare the framebuffers for the planes as follows:
  - The overlay plane contains general desktop UI, video player controls, and video subtitles
  - Primary plane contains one or more videos

.. note:: Keep in mind that we could extend this configuration to more planes,
   but that is currently not supported by our driver yet (maybe if we have a
   userspace request in the future, we can change that).

See below a single-video example:

.. kernel-figure:: single-display-mpo.svg

.. note:: We could extend this behavior to more planes, but that is currently
   not supported by our driver.

The video buffer should be used directly for the primary plane. The video can
be scaled and positioned for the desktop using the properties: CRTC_X, CRTC_Y,
CRTC_W, and CRTC_H. The primary plane should also have the color encoding and
color range properties set based on the source content:

* ``COLOR_RANGE``, ``COLOR_ENCODING``

The overlay plane should be the native size of the CRTC. The compositor must
draw a transparent cutout for where the video should be placed on the desktop
(i.e., set the alpha to zero). The primary plane video will be visible through
the underlay. The overlay plane's buffer may remain static while the primary
plane's framebuffer is used for standard double-buffered playback.

The compositor should create a YUV buffer matching the native size of the CRTC.
Each video buffer should be composited onto this YUV buffer for direct YUV
scanout. The primary plane should have the color encoding and color range
properties set based on the source content: ``COLOR_RANGE``,
``COLOR_ENCODING``. However, be mindful that the source color space and
encoding match for each video since it affect the entire plane.

The overlay plane should be the native size of the CRTC. The compositor must
draw a transparent cutout for where each video should be placed on the desktop
(i.e., set the alpha to zero). The primary plane videos will be visible through
the underlay. The overlay plane's buffer may remain static while compositing
operations for video playback will be done on the video buffer.

This kernel interface is validated using IGT GPU Tools. The following tests can
be run to validate positioning, blending, scaling under a variety of sequences
and interactions with operations such as DPMS and S3:

- ``kms_plane@plane-panning-bottom-right-pipe-*-planes``
- ``kms_plane@plane-panning-bottom-right-suspend-pipe-*-``
- ``kms_plane@plane-panning-top-left-pipe-*-``
- ``kms_plane@plane-position-covered-pipe-*-``
- ``kms_plane@plane-position-hole-dpms-pipe-*-``
- ``kms_plane@plane-position-hole-pipe-*-``
- ``kms_plane_multiple@atomic-pipe-*-tiling-``
- ``kms_plane_scaling@pipe-*-plane-scaling``
- ``kms_plane_alpha_blend@pipe-*-alpha-basic``
- ``kms_plane_alpha_blend@pipe-*-alpha-transparant-fb``
- ``kms_plane_alpha_blend@pipe-*-alpha-opaque-fb``
- ``kms_plane_alpha_blend@pipe-*-constant-alpha-min``
- ``kms_plane_alpha_blend@pipe-*-constant-alpha-mid``
- ``kms_plane_alpha_blend@pipe-*-constant-alpha-max``

Multiple-display MPO와 compositor policy

AMDGPU는 multiple-display MPO를 지원하지만 실제 동작은 compositor policy에 크게 의존합니다. Userspace는 video playback plane 보호 같은 정책을 정할 수 있으며, single display에서는 적었던 제약이 video가 여러 display에 걸치면 크게 늘어납니다.

`multi-display-hdcp-mpo.svg`는 video가 두 display 중앙에 걸친 경우를 보여 줍니다. Kernel이 하나의 보편 정책을 강제하는 것이 아니라 compositor가 보호 plane, 사용 가능한 pipe와 display topology를 고려해 처리 방법을 결정해야 합니다.

Multiple Display MPO
--------------------

AMDGPU supports display MPO when using multiple displays; however, this feature
behavior heavily relies on the compositor implementation. Keep in mind that
userspace can define different policies. For example, some OSes can use MPO to
protect the plane that handles the video playback; notice that we don't have
many limitations for a single display. Nonetheless, this manipulation can have
many more restrictions for a multi-display scenario. The below example shows a
video playback in the middle of two displays, and it is up to the compositor to
define a policy on how to handle it:

.. kernel-figure:: multi-display-hdcp-mpo.svg

Let's discuss some of the hardware limitations we have when dealing with
multi-display with MPO.

Pipe budget, scaling과 최소 크기

Hardware 제한을 설명하기 위해 원문은 두 display 사이를 이동하는 video를 가정합니다. Display 하나마다 최소 pipe 1개가 필요하고 MPO plane에도 추가 pipe가 필요합니다. Video가 두 display에 걸치면 양쪽에 MPO 처리가 필요해 MPO가 pipe 2개를 소비할 수 있습니다.

MPO scaling ratio는 0.25x 미만과 16x 초과를 처리할 수 없습니다. 예를 들어 3840x2160 4K video의 window는 1/4 크기인 960x540보다 작아질 수 없습니다. 이 scaling 제한은 ASIC마다 달라질 수 있습니다.

Limitations
~~~~~~~~~~~

For simplicity's sake, for discussing the hardware limitation, this
documentation supposes an example where we have two displays and video playback
that will be moved around different displays.

* **Hardware limitations**

From the DCN overview page, each display requires at least one pipe and each
MPO plane needs another pipe. As a result, when the video is in the middle of
the two displays, we need to use 2 pipes. See the example below where we avoid
pipe split:

- 1 display (1 pipe) + MPO (1 pipe), we will use two pipes
- 2 displays (2 pipes) + MPO (1-2 pipes); we will use 4 pipes. MPO in the
  middle of both displays needs 2 pipes.
- 3 Displays (3 pipes) + MPO (1-2 pipes), we need 5 pipes.

If we use MPO with multiple displays, the userspace has to decide to enable
multiple MPO by the price of limiting the number of external displays supported
or disable it in favor of multiple displays; it is a policy decision. For
example:

* When ASIC has 3 pipes, AMD hardware can NOT support 2 displays with MPO
* When ASIC has 4 pipes, AMD hardware can NOT support 3 displays with MPO

Let's briefly explore how userspace can handle these two display configurations
on an ASIC that only supports three pipes. We can have:

.. kernel-figure:: multi-display-hdcp-mpo-less-pipe-ex.svg

- Total pipes are 3
- User lights up 2 displays (2 out of 3 pipes are used)
- User launches video (1 pipe used for MPO)
- Now, if the user moves the video in the middle of 2 displays, one part of the
  video won't be MPO since we have used 3/3 pipes.

* **Scaling limitation**

MPO cannot handle scaling less than 0.25 and more than x16. For example:

If 4k video (3840x2160) is playing in windowed mode, the physical size of the
window cannot be smaller than (960x540).

.. note:: These scaling limitations might vary from ASIC to ASIC.

* **Size Limitation**

The minimum MPO size is 12px.