Display Core Debug tools

Dmesg로 Display Core 범위 좁히기

화면에 보이는 증상이라도 원인이 Display Core라는 보장은 없습니다. AMDGPU는 `soc`, `gmc`, `ih`, `psp`, `smu`, `dm`, `gfx`, `sdma`, `vcn`, `jpeg`, `mes` 같은 IP block을 순서대로 load하므로 먼저 dmesg에서 어느 component까지 초기화됐는지 확인해야 합니다.

예시에서 `<dm>`은 Display Manager이며 IP block number 5로 load됩니다. 이 줄이 없으면 Display component보다 앞선 component가 실패했을 수 있으므로 display issue로 단정하지 않습니다.

[    4.254295] [drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x744C 0x1002:0x0E3B 0xC8).
[    4.254718] [drm] register mmio base: 0xFCB00000
[    4.254918] [drm] register mmio size: 1048576
[    4.260095] [drm] add ip block number 0 <soc21_common>
[    4.260318] [drm] add ip block number 1 <gmc_v11_0>
[    4.260510] [drm] add ip block number 2 <ih_v6_0>
[    4.260696] [drm] add ip block number 3 <psp>
[    4.260878] [drm] add ip block number 4 <smu>
[    4.261057] [drm] add ip block number 5 <dm>
[    4.261231] [drm] add ip block number 6 <gfx_v11_0>
[    4.261402] [drm] add ip block number 7 <sdma_v6_0>
[    4.261568] [drm] add ip block number 8 <vcn_v4_0>
[    4.261729] [drm] add ip block number 9 <jpeg_v4_0>
[    4.261887] [drm] add ip block number 10 <mes_v11_0>

DM load를 확인한 뒤 `display core` log에서 software DC version과 hardware generation인 DCN version을 구분합니다. DC version은 매주 release되는 display code의 good/bad 지점을 좁힐 때 유용하고, DCN version은 조사할 hardware-generation directory의 단서입니다. 다만 서로 다른 DCN이 code와 callback을 재사용하므로 DCN은 안내선으로만 사용합니다.

dmesg | grep -i 'display core'

[    4.655828] [drm] Display Core v3.2.285 initialized on DCN 3.2

Bug report에는 board firmware 식별을 위해 ATOM BIOS code도 포함하는 것이 좋습니다.

dmesg  | grep -i 'ATOM BIOS'

[    4.274534] amdgpu: ATOM BIOS: 113-D7020100-102

========================
Display Core Debug tools
========================

In this section, you will find helpful information on debugging the amdgpu
driver from the display perspective. This page introduces debug mechanisms and
procedures to help you identify if some issues are related to display code.

Narrow down display issues
==========================

Since the display is the driver's visual component, it is common to see users
reporting issues as a display when another component causes the problem. This
section equips users to determine if a specific issue was caused by the display
component or another part of the driver.

DC dmesg important messages
---------------------------

The dmesg log is the first source of information to be checked, and amdgpu
takes advantage of this feature by logging some valuable information. When
looking for the issues associated with amdgpu, remember that each component of
the driver (e.g., smu, PSP, dm, etc.) is loaded one by one, and this
information can be found in the dmesg log. In this sense, look for the part of
the log that looks like the below log snippet::

  [    4.254295] [drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x744C 0x1002:0x0E3B 0xC8).
  [    4.254718] [drm] register mmio base: 0xFCB00000
  [    4.254918] [drm] register mmio size: 1048576
  [    4.260095] [drm] add ip block number 0 <soc21_common>
  [    4.260318] [drm] add ip block number 1 <gmc_v11_0>
  [    4.260510] [drm] add ip block number 2 <ih_v6_0>
  [    4.260696] [drm] add ip block number 3 <psp>
  [    4.260878] [drm] add ip block number 4 <smu>
  [    4.261057] [drm] add ip block number 5 <dm>
  [    4.261231] [drm] add ip block number 6 <gfx_v11_0>
  [    4.261402] [drm] add ip block number 7 <sdma_v6_0>
  [    4.261568] [drm] add ip block number 8 <vcn_v4_0>
  [    4.261729] [drm] add ip block number 9 <jpeg_v4_0>
  [    4.261887] [drm] add ip block number 10 <mes_v11_0>

From the above example, you can see the line that reports that `<dm>`,
(**Display Manager**), was loaded, which means that display can be part of the
issue. If you do not see that line, something else might have failed before
amdgpu loads the display component, indicating that we don't have a
display issue.

After you identified that the DM was loaded correctly, you can check for the
display version of the hardware in use, which can be retrieved from the dmesg
log with the command::

  dmesg | grep -i 'display core'

This command shows a message that looks like this::

  [    4.655828] [drm] Display Core v3.2.285 initialized on DCN 3.2

This message has two key pieces of information:

* **The DC version (e.g., v3.2.285)**: Display developers release a new DC version
  every week, and this information can be advantageous in a situation where a
  user/developer must find a good point versus a bad point based on a tested
  version of the display code. Remember from page :ref:`Display Core <amdgpu-display-core>`,
  that every week the new patches for display are heavily tested with IGT and
  manual tests.
* **The DCN version (e.g., DCN 3.2)**: The DCN block is associated with the
  hardware generation, and the DCN version conveys the hardware generation that
  the driver is currently running. This information helps to narrow down the
  code debug area since each DCN version has its files in the DC folder per DCN
  component (from the example, the developer might want to focus on
  files/folders/functions/structs with the dcn32 label might be executed).
  However, keep in mind that DC reuses code across different DCN versions; for
  example, it is expected to have some callbacks set in one DCN that are the same
  as those from another DCN. In summary, use the DCN version just as a guide.

From the dmesg file, it is also possible to get the ATOM bios code by using::

  dmesg  | grep -i 'ATOM BIOS'

Which generates an output that looks like this::

  [    4.274534] amdgpu: ATOM BIOS: 113-D7020100-102

This type of information is useful to be reported.

Display Core를 제외해 원인 분리하기

단순한 재현 시나리오에서 display가 원인이 아니라고 의심되면 `dm` IP block을 mask에서 제외해 DC 없이 boot하는 비교 실험을 할 수 있습니다. 먼저 dmesg의 `add ip block number`에서 hardware별 DM ID를 찾습니다.

Mask는 `0xffffffff`에서 DM ID에 해당하는 bit를 지웁니다. 예시 DM ID 5에서는 `0xffffffdf`가 되고 bootloader parameter `amdgpu.ip_block_mask`에 설정합니다.

0xffffffff & ~(1 << [DM ID])

0xffffffff & ~(1 << 5) = 0xffffffdf

amdgpu.ip_block_mask = 0xffffffdf

DC를 꺼도 문제가 남으면 DC를 원인 후보에서 제외할 수 있습니다. 문제가 사라지면 DC 관련 가능성을 계속 좁혀야 합니다. Game처럼 display component가 재현에 필수인 경우 이 방법을 쓸 수 없으며, DC 비활성화는 display output이 없어질 가능성이 큽니다.

Avoid loading display core
--------------------------

Sometimes, it might be hard to figure out which part of the driver is causing
the issue; if you suspect that the display is not part of the problem and your
bug scenario is simple (e.g., some desktop configuration) you can try to remove
the display component from the equation. First, you need to identify `dm` ID
from the dmesg log; for example, search for the following log::

  [    4.254295] [drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x744C 0x1002:0x0E3B 0xC8).
  [..]
  [    4.260095] [drm] add ip block number 0 <soc21_common>
  [    4.260318] [drm] add ip block number 1 <gmc_v11_0>
  [..]
  [    4.261057] [drm] add ip block number 5 <dm>

Notice from the above example that the `dm` id is 5 for this specific hardware.
Next, you need to run the following binary operation to identify the IP block
mask::

  0xffffffff & ~(1 << [DM ID])

From our example the IP mask is::

 0xffffffff & ~(1 << 5) = 0xffffffdf

Finally, to disable DC, you just need to set the below parameter in your
bootloader::

 amdgpu.ip_block_mask = 0xffffffdf

If you can boot your system with the DC disabled and still see the issue, it
means you can rule DC out of the equation. However, if the bug disappears, you
still need to consider the DC part of the problem and keep narrowing down the
issue. In some scenarios, disabling DC is impossible since it might be
necessary to use the display component to reproduce the issue (e.g., play a
game).

**Note: This will probably lead to the absence of a display output.**

Flicker와 artifact 원인 분류

Display flickering은 GPU 전력 부족이나 DPM switch 문제일 수 있습니다. 첫 일반 검증으로 GPU/APU를 허용된 최대 power level인 `high`에 고정하면 DPM 전환을 끌 수 있습니다.

bash -c "echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level"

Artifact는 localized와 general로 구분합니다. UI window corner 같은 특정 영역의 localized artifact는 Mesa 등 userspace 가능성이 큽니다. 전체 화면의 general artifact는 display parameter misconfiguration 또는 userspace rendering 어느 쪽에서도 발생할 수 있습니다.

Display flickering
------------------

Display flickering might have multiple causes; one is the lack of proper power
to the GPU or problems in the DPM switches. A good first generic verification
is to set the GPU to use high voltage::

   bash -c "echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level"

The above command sets the GPU/APU to use the maximum power allowed which
disables DPM switches. If forcing DPM levels high does not fix the issue, it
is less likely that the issue is related to power management. If the issue
disappears, there is a good chance that other components might be involved, and
the display should not be ignored since this could be a DPM issues. From the
display side, if the power increase fixes the issue, it is worth debugging the
clock configuration and the pipe split police used in the specific
configuration.

Display artifacts
-----------------

Users may see some screen artifacts that can be categorized into two different
types: localized artifacts and general artifacts. The localized artifacts
happen in some specific areas, such as around the UI window corners; if you see
this type of issue, there is a considerable chance that you have a userspace
problem, likely Mesa or similar. The general artifacts usually happen on the
entire screen. They might be caused by a misconfiguration at the driver level
of the display parameters, but the userspace might also cause this issue. One
way to identify the source of the problem is to take a screenshot or make a
desktop video capture when the problem happens; after checking the
screenshot/video recording, if you don't see any of the artifacts, it means
that the issue is likely on the driver side. If you can still see the
problem in the data collected, it is an issue that probably happened during
rendering, and the display code just got the framebuffer already corrupted.

dc_debug_options로 기능 단위 분리

각 DCE/DCN component는 hardware 특성에 맞춰 `dc_debug_options` struct를 정적으로 초기화합니다. Bring-up 단계에서 여러 기능을 끈 채 시작해 하나씩 enable할 수 있고, 특정 bug를 조사할 때도 기능별 비교 실험에 사용합니다.

화면 일부에 수평 띠 형태의 flicker가 나타나면 Sub-Viewport 문제일 수 있습니다. 대상 DCN의 정적 `dc_debug_options`에서 `force_disable_subvp`를 true로 설정해 변화를 확인하고, 같은 방식으로 `fams2_config`와 `enable_single_display_2to1_odm_policy`를 꺼 비교할 수 있습니다.

Disabling/Enabling specific features
====================================

DC has a struct named `dc_debug_options`, which is statically initialized by
all DCE/DCN components based on the specific hardware characteristic. This
structure usually facilitates the bring-up phase since developers can start
with many disabled features and enable them individually. This is also an
important debug feature since users can change it when debugging specific
issues.

For example, dGPU users sometimes see a problem where a horizontal fillet of
flickering happens in some specific part of the screen. This could be an
indication of Sub-Viewport issues; after the users identified the target DCN,
they can set the `force_disable_subvp` field to true in the statically
initialized version of `dc_debug_options` to see if the issue gets fixed. Along
the same lines, users/developers can also try to turn off `fams2_config` and
`enable_single_display_2to1_odm_policy`. In summary, the `dc_debug_options` is
an interesting form for identifying the problem.

Visual Confirmation으로 plane과 pipe split 확인

Visual Confirmation은 driver가 scanout 시점에 색상 bar를 추가해 hardware plane과 pipe 정보를 화면에 표시하는 기능입니다. 일반 interface는 `amdgpu_dm_visual_confirm`에 scenario 번호 `N`을 write하는 것입니다.

echo <N> > /sys/kernel/debug/dri/0/amdgpu_dm_visual_confirm

Multiple Planes debug에서는 값 1을 쓰고 GUI를 reload합니다. Plane configuration이 바뀌거나 full update가 발생하면 각 hardware plane 아래쪽에 bar가 나타납니다.

echo 1 > /sys/kernel/debug/dri/0/amdgpu_dm_visual_confirm

정상 상태에서는 visual corruption, underflow, screen flash, black screen, cursor corruption이 없어야 합니다. Window transition이나 resize 중에는 multiple plane이 잠깐 disable될 수 있지만 작업이 끝나면 돌아와야 합니다.

Pipe Split debug에서도 값 1을 사용합니다. Split이 있으면 display 전체 폭을 덮는 작은 red bar와 두 번째 pipe를 덮는 추가 bar가 보여 두 번째 pipe 영역의 bar가 더 높게 보입니다.

DC Visual Confirmation
======================

Display core provides a feature named visual confirmation, which is a set of
bars added at the scanout time by the driver to convey some specific
information. In general, you can enable this debug option by using::

  echo <N> > /sys/kernel/debug/dri/0/amdgpu_dm_visual_confirm

Where `N` is an integer number for some specific scenarios that the developer
wants to enable, you will see some of these debug cases in the following
subsection.

Multiple Planes Debug
---------------------

If you want to enable or debug multiple planes in a specific user-space
application, you can leverage a debug feature named visual confirm. For
enabling it, you will need::

  echo 1 > /sys/kernel/debug/dri/0/amdgpu_dm_visual_confirm

You need to reload your GUI to see the visual confirmation. When the plane
configuration changes or a full update occurs there will be a colored bar at
the bottom of each hardware plane being drawn on the screen.

* The color indicates the format - For example, red is AR24 and green is NV12
* The height of the bar indicates the index of the plane
* Pipe split can be observed if there are two bars with a difference in height
  covering the same plane

Consider the video playback case in which a video is played in a specific
plane, and the desktop is drawn in another plane. The video plane should
feature one or two green bars at the bottom of the video depending on pipe
split configuration.

* There should **not** be any visual corruption
* There should **not** be any underflow or screen flashes
* There should **not** be any black screens
* There should **not** be any cursor corruption
* Multiple plane **may** be briefly disabled during window transitions or
  resizing but should come back after the action has finished

Pipe Split Debug
----------------

Sometimes we need to debug if DCN is splitting pipes correctly, and visual
confirmation is also handy for this case. Similar to the MPO case, you can use
the below command to enable visual confirmation::

  echo 1 > /sys/kernel/debug/dri/0/amdgpu_dm_visual_confirm

In this case, if you have a pipe split, you will see one small red bar at the
bottom of the display covering the entire display width and another bar
covering the second pipe. In other words, you will see a bit high bar in the
second pipe.

DTN log와 display firmware 정보 수집

Display Test Next(DTN) log는 DCN hardware configuration의 여러 상태 값을 debugfs로 dump합니다. 현재 snapshot을 읽거나 `watch -d`로 상태 변화를 실시간 비교할 수 있습니다.

cat /sys/kernel/debug/dri/0/amdgpu_dm_dtn_log

sudo watch -d cat /sys/kernel/debug/dri/0/amdgpu_dm_dtn_log

전체 firmware version은 `amdgpu_firmware_info`에서 수집하며 display 관점에서는 DMCU와 DMCUB firmware에 특히 주의합니다.

cat /sys/kernel/debug/dri/0/amdgpu_firmware_info

DTN Debug
=========

DC (DCN) provides an extensive log that dumps multiple details from our
hardware configuration. Via debugfs, you can capture those status values by
using Display Test Next (DTN) log, which can be captured via debugfs by using::

  cat /sys/kernel/debug/dri/0/amdgpu_dm_dtn_log

Since this log is updated accordingly with DCN status, you can also follow the
change in real-time by using something like::

  sudo watch -d cat /sys/kernel/debug/dri/0/amdgpu_dm_dtn_log

When reporting a bug related to DC, consider attaching this log before and
after you reproduce the bug.

Collect Firmware information
============================

When reporting issues, it is important to have the firmware information since
it can be helpful for debugging purposes. To get all the firmware information,
use the command::

  cat /sys/kernel/debug/dri/0/amdgpu_firmware_info

From the display perspective, pay attention to the firmware of the DMCU and
DMCUB.

DMUB firmware trace group과 수집 절차

기능이 주로 DMUB firmware에 구현돼 있으면 dmesg에는 일반 timeout만 남을 수 있습니다. 이때 `amdgpu_dm_dmub_trace_mask`의 관련 bit를 enable해 DMUB command를 trace합니다. 모든 ASIC이 아래 모든 group을 지원하는 것은 아닙니다.

Name, Mask Value
INFO, 0x1
IRQ SVC, 0x2
VBIOS, 0x4
REGISTER, 0x8
PHY DBG, 0x10
PSR, 0x20
AUX, 0x40
SMU, 0x80
MALL, 0x100
ABM, 0x200
ALPM, 0x400
TIMER, 0x800
HW LOCK MGR, 0x1000
INBOX1, 0x2000
PHY SEQ, 0x4000
PSR STATE, 0x8000
ZSTATE, 0x10000
TRANSMITTER CTL, 0x20000
PANEL CNTL, 0x40000
FAMS, 0x80000
DPIA, 0x100000
SUBVP, 0x200000
INBOX0, 0x400000
SDP, 0x4000000
REPLAY, 0x8000000
REPLAY RESIDENCY, 0x20000000
CURSOR INFO, 0x80000000
IPS, 0x100000000

PSR만 trace하는 원문 예시는 `0x8020`을 사용합니다. 이는 PSR `0x20`과 PSR STATE `0x8000` group을 함께 선택한 값입니다.

# echo 0x8020 > /sys/kernel/debug/dri/0/amdgpu_dm_dmub_trace_mask

그 다음 DMCUB trace event buffer logging을 enable하고 문제를 재현합니다. 재현 뒤 logging을 disable한 다음 trace buffer를 읽습니다.

# echo 1 > /sys/kernel/debug/dri/0/amdgpu_dm_dmcub_trace_event_en

# echo 0 > /sys/kernel/debug/dri/0/amdgpu_dm_dmcub_trace_event_en
# cat /sys/kernel/debug/dri/0/amdgpu_dm_dmub_tracebuffer

DMUB Firmware Debug
===================

Sometimes, dmesg logs aren't enough. This is especially true if a feature is
implemented primarily in DMUB firmware. In such cases, all we see in dmesg when
an issue arises is some generic timeout error. So, to get more relevant
information, we can trace DMUB commands by enabling the relevant bits in
`amdgpu_dm_dmub_trace_mask`.

Currently, we support the tracing of the following groups:

Trace Groups
------------

.. csv-table::
   :header-rows: 1
   :widths: 1, 1
   :file: ./trace-groups-table.csv

**Note: Not all ASICs support all of the listed trace groups**

So, to enable just PSR tracing you can use the following command::

  # echo 0x8020 > /sys/kernel/debug/dri/0/amdgpu_dm_dmub_trace_mask

Then, you need to enable logging trace events to the buffer, which you can do
using the following::

  # echo 1 > /sys/kernel/debug/dri/0/amdgpu_dm_dmcub_trace_event_en

Lastly, after you are able to reproduce the issue you are trying to debug,
you can disable tracing and read the trace log by using the following::

  # echo 0 > /sys/kernel/debug/dri/0/amdgpu_dm_dmcub_trace_event_en
  # cat /sys/kernel/debug/dri/0/amdgpu_dm_dmub_tracebuffer

So, when reporting bugs related to features such as PSR and ABM, consider
enabling the relevant bits in the mask before reproducing the issue and
attach the log that you obtain from the trace buffer in any bug reports that you
create.