← Documents Documentation/arch/x86/amd-debugging.rst GitHub 원문 ↗

Linux 6.18.37 · Architecture

Debugging AMD Zen Systems

AMD Zen의 S3/s2idle suspend, IRQ·GPIO·ACPI wakeup 추적, 과거 resume 회귀, runtime power, BIOS AML trace, persistent reboot reason을 진단하는 안내서입니다.

Source pathDocumentation/arch/x86/amd-debugging.rst
Source versionLinux v6.18.37
TranslationDUJINLABS 전문 번역 + 해설

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

요약과 해설

amd-debugging.rst:1-368

AMD system은 S3와 s2idle 중 하나를 사용하며 `/sys/power/mem_sleep`으로 mode를 판별합니다. s2idle 문제는 `/sys/power/pm_wakeup_irq`, `/proc/interrupts`, PM debug sysfs, ACPI trace를 차례로 확인하고 GPIO source는 `_EVT` entry에서 ACPI path와 PCI `physical_node`까지 연결해 추적합니다.

문서는 core offlining, DMCUB resume corruption, back-to-back suspend, HPET wakeup, StorageD3Enable 누락, IRQ1 firmware bug, mailbox timeout, panel HPD interrupt 등 실제 수정 사례와 해당 upstream commit을 제공합니다. runtime power에서는 vendor BIOS의 ASPM 설정, L1.2, CPU EPP policy가 중요합니다.

serial UART가 없는 OEM machine에서는 `CONFIG_ACPI_DEBUG`와 ACPICA `\M460` tracing으로 BIOS AML argument를 kernel ring buffer에 남길 수 있습니다. random reboot 뒤에는 다음 boot까지 보존되는 register bit를 읽어 pin, software, ACPI, hardware, remote, internal CPU 원인을 구분합니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기
1 .. SPDX-License-Identifier: GPL-2.0
2
3 Debugging AMD Zen systems
4 +++++++++++++++++++++++++
5
6 Introduction
7 ============
8
9 This document describes techniques that are useful for debugging issues with
10 AMD Zen systems. It is intended for use by developers and technical users
11 to help identify and resolve issues.
12
13 S3 vs s2idle
14 ============
15
16 On AMD systems, it's not possible to simultaneously support suspend-to-RAM (S3)
17 and suspend-to-idle (s2idle). To confirm which mode your system supports you
18 can look at ``cat /sys/power/mem_sleep``. If it shows ``s2idle [deep]`` then
19 *S3* is supported. If it shows ``[s2idle]`` then *s2idle* is
20 supported.
21
22 On systems that support *S3*, the firmware will be utilized to put all hardware into
23 the appropriate low power state.
24
25 On systems that support *s2idle*, the kernel will be responsible for transitioning devices
26 into the appropriate low power state. When all devices are in the appropriate low
27 power state, the hardware will transition into a hardware sleep state.
28
29 After a suspend cycle you can tell how much time was spent in a hardware sleep
30 state by looking at ``cat /sys/power/suspend_stats/last_hw_sleep``.
31
32 This flowchart explains how the AMD s2idle suspend flow works.
33
34 .. kernel-figure:: suspend.svg
35
36 This flowchart explains how the amd s2idle resume flow works.
37
38 .. kernel-figure:: resume.svg
39
40 s2idle debugging tool
41 =====================
42
43 As there are a lot of places that problems can occur, a debugging tool has been
44 created at
45 `amd-debug-tools <https://git.kernel.org/pub/scm/linux/kernel/git/superm1/amd-debug-tools.git/about/>`_
46 that can help test for common problems and offer suggestions.
47
48 If you have an s2idle issue, it's best to start with this and follow instructions
49 from its findings. If you continue to have an issue, raise a bug with the
50 report generated from this script to
51 `drm/amd gitlab <https://gitlab.freedesktop.org/drm/amd/-/issues/new?issuable_template=s2idle_BUG_TEMPLATE>`_.
52
53 Spurious s2idle wakeups from an IRQ
54 ===================================
55
56 Spurious wakeups will generally have an IRQ set to ``/sys/power/pm_wakeup_irq``.
57 This can be matched to ``/proc/interrupts`` to determine what device woke the system.
58
59 If this isn't enough to debug the problem, then the following sysfs files
60 can be set to add more verbosity to the wakeup process: ::
61
62 # echo 1 | sudo tee /sys/power/pm_debug_messages
63 # echo 1 | sudo tee /sys/power/pm_print_times
64
65 After making those changes, the kernel will display messages that can
66 be traced back to kernel s2idle loop code as well as display any active
67 GPIO sources while waking up.
68
69 If the wakeup is caused by the ACPI SCI, additional ACPI debugging may be
70 needed. These commands can enable additional trace data: ::
71
72 # echo enable | sudo tee /sys/module/acpi/parameters/trace_state
73 # echo 1 | sudo tee /sys/module/acpi/parameters/aml_debug_output
74 # echo 0x0800000f | sudo tee /sys/module/acpi/parameters/debug_level
75 # echo 0xffff0000 | sudo tee /sys/module/acpi/parameters/debug_layer
76
77 Spurious s2idle wakeups from a GPIO
78 ===================================
79
80 If a GPIO is active when waking up the system ideally you would look at the
81 schematic to determine what device it is associated with. If the schematic
82 is not available, another tactic is to look at the ACPI _EVT() entry
83 to determine what device is notified when that GPIO is active.
84
85 For a hypothetical example, say that GPIO 59 woke up the system. You can
86 look at the SSDT to determine what device is notified when GPIO 59 is active.
87
88 First convert the GPIO number into hex. ::
89
90 $ python3 -c "print(hex(59))"
91 0x3b
92
93 Next determine which ACPI table has the ``_EVT`` entry. For example: ::
94
95 $ sudo grep EVT /sys/firmware/acpi/tables/SSDT*
96 grep: /sys/firmware/acpi/tables/SSDT27: binary file matches
97
98 Decode this table::
99
100 $ sudo cp /sys/firmware/acpi/tables/SSDT27 .
101 $ sudo iasl -d SSDT27
102
103 Then look at the table and find the matching entry for GPIO 0x3b. ::
104
105 Case (0x3B)
106 {
107 M000 (0x393B)
108 M460 (" Notify (\\_SB.PCI0.GP17.XHC1, 0x02)\n", Zero, Zero, Zero, Zero, Zero, Zero)
109 Notify (\_SB.PCI0.GP17.XHC1, 0x02) // Device Wake
110 }
111
112 You can see in this case that the device ``\_SB.PCI0.GP17.XHC1`` is notified
113 when GPIO 59 is active. It's obvious this is an XHCI controller, but to go a
114 step further you can figure out which XHCI controller it is by matching it to
115 ACPI.::
116
117 $ grep "PCI0.GP17.XHC1" /sys/bus/acpi/devices/*/path
118 /sys/bus/acpi/devices/device:2d/path:\_SB_.PCI0.GP17.XHC1
119 /sys/bus/acpi/devices/device:2e/path:\_SB_.PCI0.GP17.XHC1.RHUB
120 /sys/bus/acpi/devices/device:2f/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT1
121 /sys/bus/acpi/devices/device:30/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT1.CAM0
122 /sys/bus/acpi/devices/device:31/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT1.CAM1
123 /sys/bus/acpi/devices/device:32/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT2
124 /sys/bus/acpi/devices/LNXPOWER:0d/path:\_SB_.PCI0.GP17.XHC1.PWRS
125
126 Here you can see it matches to ``device:2d``. Look at the ``physical_node``
127 to determine what PCI device that actually is. ::
128
129 $ ls -l /sys/bus/acpi/devices/device:2d/physical_node
130 lrwxrwxrwx 1 root root 0 Feb 12 13:22 /sys/bus/acpi/devices/device:2d/physical_node -> ../../../../../pci0000:00/0000:00:08.1/0000:c2:00.4
131
132 So there you have it: the PCI device associated with this GPIO wakeup was ``0000:c2:00.4``.
133
134 The ``amd_s2idle.py`` script will capture most of these artifacts for you.
135
136 s2idle PM debug messages
137 ========================
138
139 During the s2idle flow on AMD systems, the ACPI LPS0 driver is responsible
140 to check all uPEP constraints. Failing uPEP constraints does not prevent
141 s0i3 entry. This means that if some constraints are not met, it is possible
142 the kernel may attempt to enter s2idle even if there are some known issues.
143
144 To activate PM debugging, either specify ``pm_debug_messagess`` kernel
145 command-line option at boot or write to ``/sys/power/pm_debug_messages``.
146 Unmet constraints will be displayed in the kernel log and can be
147 viewed by logging tools that process kernel ring buffer like ``dmesg`` or
148 ``journalctl``."
149
150 If the system freezes on entry/exit before these messages are flushed, a
151 useful debugging tactic is to unbind the ``amd_pmc`` driver to prevent
152 notification to the platform to start s0i3 entry. This will stop the
153 system from freezing on entry or exit and let you view all the failed
154 constraints. ::
155
156 cd /sys/bus/platform/drivers/amd_pmc
157 ls | grep AMD | sudo tee unbind
158
159 After doing this, run the suspend cycle and look specifically for errors around: ::
160
161 ACPI: LPI: Constraint not met; min power state:%s current power state:%s
162
163 Historical examples of s2idle issues
164 ====================================
165
166 To help understand the types of issues that can occur and how to debug them,
167 here are some historical examples of s2idle issues that have been resolved.
168
169 Core offlining
170 --------------
171 An end user had reported that taking a core offline would prevent the system
172 from properly entering s0i3. This was debugged using internal AMD tools
173 to capture and display a stream of metrics from the hardware showing what changed
174 when a core was offlined. It was determined that the hardware didn't get
175 notification the offline cores were in the deepest state, and so it prevented
176 CPU from going into the deepest state. The issue was debugged to a missing
177 command to put cores into C3 upon offline.
178
179 `commit d6b88ce2eb9d2 ("ACPI: processor idle: Allow playing dead in C3 state") <https://git.kernel.org/torvalds/c/d6b88ce2eb9d2>`_
180
181 Corruption after resume
182 -----------------------
183 A big problem that occurred with Rembrandt was that there was graphical
184 corruption after resume. This happened because of a misalignment of PSP
185 and driver responsibility. The PSP will save and restore DMCUB, but the
186 driver assumed it needed to reset DMCUB on resume.
187 This actually was a misalignment for earlier silicon as well, but was not
188 observed.
189
190 `commit 79d6b9351f086 ("drm/amd/display: Don't reinitialize DMCUB on s0ix resume") <https://git.kernel.org/torvalds/c/79d6b9351f086>`_
191
192 Back to Back suspends fail
193 --------------------------
194 When using a wakeup source that triggers the IRQ to wakeup, a bug in the
195 pinctrl-amd driver may capture the wrong state of the IRQ and prevent the
196 system going back to sleep properly.
197
198 `commit b8c824a869f22 ("pinctrl: amd: Don't save/restore interrupt status and wake status bits") <https://git.kernel.org/torvalds/c/b8c824a869f22>`_
199
200 Spurious timer based wakeup after 5 minutes
201 -------------------------------------------
202 The HPET was being used to program the wakeup source for the system, however
203 this was causing a spurious wakeup after 5 minutes. The correct alarm to use
204 was the ACPI alarm.
205
206 `commit 3d762e21d5637 ("rtc: cmos: Use ACPI alarm for non-Intel x86 systems too") <https://git.kernel.org/torvalds/c/3d762e21d5637>`_
207
208 Disk disappears after resume
209 ----------------------------
210 After resuming from s2idle, the NVME disk would disappear. This was due to the
211 BIOS not specifying the _DSD StorageD3Enable property. This caused the NVME
212 driver not to put the disk into the expected state at suspend and to fail
213 on resume.
214
215 `commit e79a10652bbd3 ("ACPI: x86: Force StorageD3Enable on more products") <https://git.kernel.org/torvalds/c/e79a10652bbd3>`_
216
217 Spurious IRQ1
218 -------------
219 A number of Renoir, Lucienne, Cezanne, & Barcelo platforms have a
220 platform firmware bug where IRQ1 is triggered during s0i3 resume.
221
222 This was fixed in the platform firmware, but a number of systems didn't
223 receive any more platform firmware updates.
224
225 `commit 8e60615e89321 ("platform/x86/amd: pmc: Disable IRQ1 wakeup for RN/CZN") <https://git.kernel.org/torvalds/c/8e60615e89321>`_
226
227 Hardware timeout
228 ----------------
229 The hardware performs many actions besides accepting the values from
230 amd-pmc driver. As the communication path with the hardware is a mailbox,
231 it's possible that it might not respond quickly enough.
232 This issue manifested as a failure to suspend: ::
233
234 PM: dpm_run_callback(): acpi_subsys_suspend_noirq+0x0/0x50 returns -110
235 amd_pmc AMDI0005:00: PM: failed to suspend noirq: error -110
236
237 The timing problem was identified by comparing the values of the idle mask.
238
239 `commit 3c3c8e88c8712 ("platform/x86: amd-pmc: Increase the response register timeout") <https://git.kernel.org/torvalds/c/3c3c8e88c8712>`_
240
241 Failed to reach hardware sleep state with panel on
242 --------------------------------------------------
243 On some Strix systems certain panels were observed to block the system from
244 entering a hardware sleep state if the internal panel was on during the sequence.
245
246 Even though the panel got turned off during suspend it exposed a timing problem
247 where an interrupt caused the display hardware to wake up and block low power
248 state entry.
249
250 `commit 40b8c14936bd2 ("drm/amd/display: Disable unneeded hpd interrupts during dm_init") <https://git.kernel.org/torvalds/c/40b8c14936bd2>`_
251
252 Runtime power consumption issues
253 ================================
254
255 Runtime power consumption is influenced by many factors, including but not
256 limited to the configuration of the PCIe Active State Power Management (ASPM),
257 the display brightness, the EPP policy of the CPU, and the power management
258 of the devices.
259
260 ASPM
261 ----
262 For the best runtime power consumption, ASPM should be programmed as intended
263 by the BIOS from the hardware vendor. To accomplish this the Linux kernel
264 should be compiled with ``CONFIG_PCIEASPM_DEFAULT`` set to ``y`` and the
265 sysfs file ``/sys/module/pcie_aspm/parameters/policy`` should not be modified.
266
267 Most notably, if L1.2 is not configured properly for any devices, the SoC
268 will not be able to enter the deepest idle state.
269
270 EPP Policy
271 ----------
272 The ``energy_performance_preference`` sysfs file can be used to set a bias
273 of efficiency or performance for a CPU. This has a direct relationship on
274 the battery life when more heavily biased towards performance.
275
276
277 BIOS debug messages
278 ===================
279
280 Most OEM machines don't have a serial UART for outputting kernel or BIOS
281 debug messages. However BIOS debug messages are useful for understanding
282 both BIOS bugs and bugs with the Linux kernel drivers that call BIOS AML.
283
284 As the BIOS on most OEM AMD systems are based off an AMD reference BIOS,
285 the infrastructure used for exporting debugging messages is often the same
286 as AMD reference BIOS.
287
288 Manually Parsing
289 ----------------
290 There is generally an ACPI method ``\M460`` that different paths of the AML
291 will call to emit a message to the BIOS serial log. This method takes
292 7 arguments, with the first being a string and the rest being optional
293 integers::
294
295 Method (M460, 7, Serialized)
296
297 Here is an example of a string that BIOS AML may call out using ``\M460``::
298
299 M460 (" OEM-ASL-PCIe Address (0x%X)._REG (%d %d) PCSA = %d\n", DADR, Arg0, Arg1, PCSA, Zero, Zero)
300
301 Normally when executed, the ``\M460`` method would populate the additional
302 arguments into the string. In order to get these messages from the Linux
303 kernel a hook has been added into ACPICA that can capture the *arguments*
304 sent to ``\M460`` and print them to the kernel ring buffer.
305 For example the following message could be emitted into kernel ring buffer::
306
307 extrace-0174 ex_trace_args : " OEM-ASL-PCIe Address (0x%X)._REG (%d %d) PCSA = %d\n", ec106000, 2, 1, 1, 0, 0
308
309 In order to get these messages, you need to compile with ``CONFIG_ACPI_DEBUG``
310 and then turn on the following ACPICA tracing parameters.
311 This can be done either on the kernel command line or at runtime:
312
313 * ``acpi.trace_method_name=\M460``
314 * ``acpi.trace_state=method``
315
316 NOTE: These can be very noisy at bootup. If you turn these parameters on
317 the kernel command, please also consider turning up ``CONFIG_LOG_BUF_SHIFT``
318 to a larger size such as 17 to avoid losing early boot messages.
319
320 Tool assisted Parsing
321 ---------------------
322 As mentioned above, parsing by hand can be tedious, especially with a lot of
323 messages. To help with this, a tool has been created at
324 `amd-debug-tools <https://git.kernel.org/pub/scm/linux/kernel/git/superm1/amd-debug-tools.git/about/>`_
325 to help parse the messages.
326
327 Random reboot issues
328 ====================
329
330 When a random reboot occurs, the high-level reason for the reboot is stored
331 in a register that will persist onto the next boot.
332
333 There are 6 classes of reasons for the reboot:
334 * Software induced
335 * Power state transition
336 * Pin induced
337 * Hardware induced
338 * Remote reset
339 * Internal CPU event
340
341 .. csv-table::
342 :header: "Bit", "Type", "Reason"
343 :align: left
344
345 "0", "Pin", "thermal pin BP_THERMTRIP_L was tripped"
346 "1", "Pin", "power button was pressed for 4 seconds"
347 "2", "Pin", "shutdown pin was tripped"
348 "4", "Remote", "remote ASF power off command was received"
349 "9", "Internal", "internal CPU thermal limit was tripped"
350 "16", "Pin", "system reset pin BP_SYS_RST_L was tripped"
351 "17", "Software", "software issued PCI reset"
352 "18", "Software", "software wrote 0x4 to reset control register 0xCF9"
353 "19", "Software", "software wrote 0x6 to reset control register 0xCF9"
354 "20", "Software", "software wrote 0xE to reset control register 0xCF9"
355 "21", "ACPI-state", "ACPI power state transition occurred"
356 "22", "Pin", "keyboard reset pin KB_RST_L was tripped"
357 "23", "Internal", "internal CPU shutdown event occurred"
358 "24", "Hardware", "system failed to boot before failed boot timer expired"
359 "25", "Hardware", "hardware watchdog timer expired"
360 "26", "Remote", "remote ASF reset command was received"
361 "27", "Internal", "an uncorrected error caused a data fabric sync flood event"
362 "29", "Internal", "FCH and MP1 failed warm reset handshake"
363 "30", "Internal", "a parity error occurred"
364 "31", "Internal", "a software sync flood event occurred"
365
366 This information is read by the kernel at bootup and printed into
367 the syslog. When a random reboot occurs this message can be helpful
368 to determine the next component to debug.
369

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

AMD Zen system debugging

1-12

이 `GPL-2.0` 문서는 AMD Zen system 문제를 debugging할 때 유용한 기법을 설명합니다. developer와 technical user가 문제를 식별하고 해결하는 데 쓰도록 작성되었습니다.

S3와 s2idle suspend 방식

13-39

AMD system은 suspend-to-RAM(S3)과 suspend-to-idle(s2idle)을 동시에 지원할 수 없습니다. `cat /sys/power/mem_sleep` 결과가 `s2idle [deep]`이면 S3를 지원하고, `[s2idle]`이면 s2idle을 지원합니다.

modelow-power 전환 책임
S3firmware가 모든 hardware를 적절한 low-power state로 전환합니다.
s2idlekernel이 device를 적절한 low-power state로 전환하고, 모두 준비되면 hardware가 hardware sleep state로 진입합니다.

suspend cycle 뒤 `cat /sys/power/suspend_stats/last_hw_sleep`을 확인하면 hardware sleep state에 머문 시간을 알 수 있습니다.

원문의 `suspend.svg`와 `resume.svg` flowchart가 설명하는 AMD s2idle suspend/resume 흐름은 다음과 같이 구조화할 수 있습니다.

AMD s2idle suspend 흐름
s2idle 요청device low-power 전환ACPI LPS0 uPEP constraint 확인platform notificationhardware sleep state

kernel이 device와 platform constraint를 준비한 뒤 hardware sleep state로 진입합니다.

AMD s2idle resume 흐름
IRQ·GPIO·ACPI wakeuphardware sleep 종료platform resume notificationdevice resumeuserspace 실행 재개

wakeup source가 hardware와 kernel의 device resume를 차례로 유발합니다.

s2idle debugging tool

40-52

s2idle 문제는 발생 지점이 많으므로 common problem을 검사하고 제안을 제공하는 `amd-debug-tools`가 `https://git.kernel.org/pub/scm/linux/kernel/git/superm1/amd-debug-tools.git/about/`에 마련되어 있습니다.

s2idle 문제가 있으면 먼저 이 tool을 실행하고 결과가 안내하는 절차를 따르는 것이 좋습니다. 문제가 계속되면 script가 생성한 report를 첨부해 `https://gitlab.freedesktop.org/drm/amd/-/issues/new?issuable_template=s2idle_BUG_TEMPLATE`의 drm/amd GitLab에 bug를 보고합니다.

IRQ와 ACPI SCI의 비정상 wakeup

53-76

spurious wakeup은 보통 `/sys/power/pm_wakeup_irq`에 IRQ를 남깁니다. 이를 `/proc/interrupts`와 대조하면 system을 깨운 device를 찾을 수 있습니다.

정보가 충분하지 않으면 다음 sysfs file을 설정해 wakeup 과정의 verbosity를 높입니다.

# echo 1 | sudo tee /sys/power/pm_debug_messages
# echo 1 | sudo tee /sys/power/pm_print_times

설정 뒤 kernel은 kernel s2idle loop code까지 추적할 수 있는 message와 wakeup 중 active한 GPIO source를 표시합니다.

ACPI SCI가 wakeup 원인이라면 추가 ACPI debugging이 필요할 수 있습니다. 다음 command로 trace state, AML debug output, debug level, debug layer를 활성화합니다.

# echo enable | sudo tee /sys/module/acpi/parameters/trace_state
# echo 1 | sudo tee /sys/module/acpi/parameters/aml_debug_output
# echo 0x0800000f | sudo tee /sys/module/acpi/parameters/debug_level
# echo 0xffff0000 | sudo tee /sys/module/acpi/parameters/debug_layer

GPIO wakeup을 ACPI와 PCI device까지 추적

77-135

wakeup 시 GPIO가 active하면 schematic에서 연결 device를 찾는 것이 가장 좋습니다. schematic이 없다면 ACPI `_EVT()` entry를 조사해 해당 GPIO가 active할 때 어떤 device가 notification을 받는지 확인할 수 있습니다.

예제로 GPIO 59가 system을 깨웠다고 가정합니다. 먼저 GPIO number를 hexadecimal `0x3b`로 변환합니다.

$ python3 -c "print(hex(59))"
0x3b

다음으로 `_EVT` entry가 들어 있는 ACPI table을 찾습니다.

$ sudo grep EVT /sys/firmware/acpi/tables/SSDT*
grep: /sys/firmware/acpi/tables/SSDT27: binary file matches

찾은 `SSDT27` table을 복사하고 `iasl`로 decode합니다.

$ sudo cp /sys/firmware/acpi/tables/SSDT27 .
$ sudo iasl -d SSDT27

decode한 table에서 GPIO `0x3B`에 대응하는 entry를 찾습니다. 이 예에서는 `\_SB.PCI0.GP17.XHC1`에 Device Wake notification을 보냅니다.

Case (0x3B)
{
    M000 (0x393B)
    M460 ("    Notify (\\_SB.PCI0.GP17.XHC1, 0x02)\n", Zero, Zero, Zero, Zero, Zero, Zero)
    Notify (\_SB.PCI0.GP17.XHC1, 0x02) // Device Wake
}

이 path는 XHCI controller임을 알 수 있습니다. 어느 XHCI controller인지 확인하려면 ACPI device path와 대조합니다.

$ grep "PCI0.GP17.XHC1" /sys/bus/acpi/devices/*/path
/sys/bus/acpi/devices/device:2d/path:\_SB_.PCI0.GP17.XHC1
/sys/bus/acpi/devices/device:2e/path:\_SB_.PCI0.GP17.XHC1.RHUB
/sys/bus/acpi/devices/device:2f/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT1
/sys/bus/acpi/devices/device:30/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT1.CAM0
/sys/bus/acpi/devices/device:31/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT1.CAM1
/sys/bus/acpi/devices/device:32/path:\_SB_.PCI0.GP17.XHC1.RHUB.PRT2
/sys/bus/acpi/devices/LNXPOWER:0d/path:\_SB_.PCI0.GP17.XHC1.PWRS

결과는 `device:2d`와 일치합니다. 실제 PCI device는 그 ACPI device의 `physical_node` symlink로 확인합니다.

$ ls -l /sys/bus/acpi/devices/device:2d/physical_node
lrwxrwxrwx 1 root root 0 Feb 12 13:22 /sys/bus/acpi/devices/device:2d/physical_node -> ../../../../../pci0000:00/0000:00:08.1/0000:c2:00.4

따라서 이 GPIO wakeup에 연결된 PCI device는 `0000:c2:00.4`입니다. `amd_s2idle.py` script가 이러한 artifact 대부분을 자동으로 수집합니다.

s2idle PM debug message와 freeze 회피

136-162

AMD s2idle 흐름에서 ACPI LPS0 driver는 모든 uPEP constraint를 확인합니다. uPEP constraint가 실패해도 s0i3 진입 자체를 막지는 않으므로 알려진 문제가 있어도 kernel이 s2idle 진입을 시도할 수 있습니다.

PM debugging은 boot 때 kernel command-line option `pm_debug_messagess`를 지정하거나 `/sys/power/pm_debug_messages`에 써서 활성화합니다. 충족되지 않은 constraint는 kernel log에 표시되며 `dmesg`나 `journalctl`처럼 kernel ring buffer를 처리하는 logging tool로 볼 수 있습니다.

원문은 kernel command-line option을 `pm_debug_messagess`로 표기합니다. sysfs file 이름은 `/sys/power/pm_debug_messages`입니다.

message가 flush되기 전에 진입 또는 이탈 과정에서 system이 freeze된다면 `amd_pmc` driver를 unbind해 platform에 s0i3 진입 시작을 알리지 않도록 할 수 있습니다. freeze를 피한 상태에서 실패한 constraint를 모두 확인합니다.

cd /sys/bus/platform/drivers/amd_pmc
ls | grep AMD | sudo tee unbind

그 뒤 suspend cycle을 실행하고 다음 error를 집중적으로 찾습니다.

ACPI: LPI: Constraint not met; min power state:%s current power state:%s

과거 사례: core offlining

163-180

다음 절들은 이미 해결된 historical s2idle issue를 통해 발생 가능한 문제 유형과 debugging 방법을 보여 줍니다.

한 user는 core를 offline하면 system이 s0i3에 제대로 진입하지 못한다고 보고했습니다. AMD 내부 tool로 hardware metric stream을 수집해 core offline 전후 차이를 확인한 결과, hardware가 offline core가 deepest state에 있다는 notification을 받지 못해 CPU의 deepest state 진입을 막고 있었습니다.

원인은 offline 시 core를 C3에 넣는 command가 빠진 것이었고 `d6b88ce2eb9d2` (`ACPI: processor idle: Allow playing dead in C3 state`) commit으로 해결했습니다: `https://git.kernel.org/torvalds/c/d6b88ce2eb9d2`.

과거 사례: resume corruption과 연속 suspend

181-199

Rembrandt에서는 resume 뒤 graphical corruption이 발생했습니다. PSP와 driver의 책임이 어긋난 것이 원인이었습니다. PSP가 DMCUB를 save/restore하는데도 driver는 resume 때 DMCUB를 reset해야 한다고 가정했습니다. earlier silicon에도 같은 불일치가 있었지만 관찰되지 않았습니다.

`79d6b9351f086` (`drm/amd/display: Don't reinitialize DMCUB on s0ix resume`) commit으로 수정했습니다: `https://git.kernel.org/torvalds/c/79d6b9351f086`.

IRQ를 trigger해 깨우는 wakeup source를 사용할 때 `pinctrl-amd` driver가 IRQ의 잘못된 state를 capture하여 system이 다시 sleep에 제대로 들어가지 못하는 bug도 있었습니다.

`b8c824a869f22` (`pinctrl: amd: Don't save/restore interrupt status and wake status bits`) commit으로 수정했습니다: `https://git.kernel.org/torvalds/c/b8c824a869f22`.

과거 사례: timer·NVMe·IRQ1 wakeup

200-226

system wakeup source를 programming할 때 HPET를 사용해 5분 뒤 spurious wakeup이 발생한 사례가 있습니다. 올바른 alarm은 ACPI alarm이었으며 `3d762e21d5637` (`rtc: cmos: Use ACPI alarm for non-Intel x86 systems too`) commit으로 수정했습니다: `https://git.kernel.org/torvalds/c/3d762e21d5637`.

s2idle resume 뒤 NVMe disk가 사라진 사례는 BIOS가 `_DSD StorageD3Enable` property를 지정하지 않았기 때문입니다. NVMe driver가 suspend 때 disk를 기대 state로 전환하지 않아 resume에 실패했습니다.

`e79a10652bbd3` (`ACPI: x86: Force StorageD3Enable on more products`) commit으로 수정했습니다: `https://git.kernel.org/torvalds/c/e79a10652bbd3`.

Renoir, Lucienne, Cezanne, Barcelo platform 다수에는 s0i3 resume 중 IRQ1을 trigger하는 firmware bug가 있었습니다. platform firmware에서 수정됐지만 더 이상 update를 받지 못한 system도 많았습니다.

`8e60615e89321` (`platform/x86/amd: pmc: Disable IRQ1 wakeup for RN/CZN`) commit이 해당 system을 보완합니다: `https://git.kernel.org/torvalds/c/8e60615e89321`.

과거 사례: mailbox timeout과 panel interrupt

227-251

hardware는 `amd-pmc` driver가 보낸 값을 받는 것 외에도 많은 작업을 수행합니다. communication path가 mailbox이므로 충분히 빨리 응답하지 못할 수 있고, 다음과 같은 suspend failure로 나타났습니다.

PM: dpm_run_callback(): acpi_subsys_suspend_noirq+0x0/0x50 returns -110
amd_pmc AMDI0005:00: PM: failed to suspend noirq: error -110

idle mask 값을 비교해 timing problem을 확인했고 `3c3c8e88c8712` (`platform/x86: amd-pmc: Increase the response register timeout`) commit으로 response-register timeout을 늘렸습니다: `https://git.kernel.org/torvalds/c/3c3c8e88c8712`.

일부 Strix system은 suspend sequence 중 internal panel이 켜져 있으면 특정 panel이 hardware sleep state 진입을 막았습니다. suspend 중 panel 자체는 꺼졌지만 interrupt가 display hardware를 다시 깨워 low-power state 진입을 차단하는 timing problem이 드러났습니다.

`40b8c14936bd2` (`drm/amd/display: Disable unneeded hpd interrupts during dm_init`) commit으로 불필요한 HPD interrupt를 비활성화했습니다: `https://git.kernel.org/torvalds/c/40b8c14936bd2`.

runtime power와 ASPM·EPP

252-275

runtime power consumption은 PCIe Active State Power Management(ASPM) 설정, display brightness, CPU EPP policy, device power management 등 많은 요인의 영향을 받습니다.

최적의 runtime power consumption을 위해 ASPM은 hardware vendor BIOS의 의도대로 programming해야 합니다. Linux kernel은 `CONFIG_PCIEASPM_DEFAULT=y`로 compile하고 `/sys/module/pcie_aspm/parameters/policy`를 변경하지 않아야 합니다. 특히 device 하나라도 L1.2가 올바르게 설정되지 않으면 SoC가 deepest idle state에 들어갈 수 없습니다.

`energy_performance_preference` sysfs file로 CPU의 efficiency 또는 performance bias를 설정할 수 있습니다. performance 쪽으로 더 치우칠수록 battery life에 직접적인 영향을 줍니다.

BIOS AML debug message 수동 해석

276-318

대부분의 OEM machine에는 kernel 또는 BIOS debug message 출력용 serial UART가 없습니다. 그러나 BIOS debug message는 BIOS bug뿐 아니라 BIOS AML을 호출하는 Linux kernel driver bug를 이해하는 데도 유용합니다.

대부분의 OEM AMD BIOS가 AMD reference BIOS에 기반하므로 debug message를 export하는 infrastructure도 흔히 동일합니다. 일반적으로 AML 여러 path가 BIOS serial log에 message를 내보낼 때 호출하는 ACPI method `\M460`이 있습니다. 이 method는 argument 7개를 받으며 첫 argument는 string이고 나머지는 optional integer입니다.

Method (M460, 7, Serialized)

BIOS AML이 `\M460`으로 출력할 수 있는 string 예제입니다.

M460 ("  OEM-ASL-PCIe Address (0x%X)._REG (%d %d)  PCSA = %d\n", DADR, Arg0, Arg1, PCSA, Zero, Zero)

정상 실행에서는 `\M460` method가 추가 argument를 format string에 채웁니다. Linux kernel에서 이 message를 얻기 위해 ACPICA에 hook을 추가해 `\M460`으로 전달된 argument를 capture하고 kernel ring buffer에 출력할 수 있습니다.

extrace-0174 ex_trace_args         :  "  OEM-ASL-PCIe Address (0x%X)._REG (%d %d)  PCSA = %d\n", ec106000, 2, 1, 1, 0, 0

message를 얻으려면 `CONFIG_ACPI_DEBUG`로 compile하고 kernel command line 또는 runtime에서 다음 ACPICA tracing parameter를 활성화합니다.

  • `acpi.trace_method_name=\M460`
  • `acpi.trace_state=method`

이 trace는 boot 때 매우 많은 message를 만들 수 있습니다. kernel command line에서 켠다면 early boot message 유실을 막도록 `CONFIG_LOG_BUF_SHIFT`를 17 같은 더 큰 값으로 높이는 것도 고려합니다.

BIOS message tool-assisted parsing

319-326

message가 많으면 수동 parsing이 번거롭습니다. `https://git.kernel.org/pub/scm/linux/kernel/git/superm1/amd-debug-tools.git/about/`의 `amd-debug-tools`가 이 message 해석을 돕습니다.

random reboot reason class

327-340

random reboot가 발생하면 reboot의 high-level reason이 다음 boot까지 유지되는 register에 저장됩니다.

reboot reason은 다음 6개 class로 나뉩니다.

  • Software induced
  • Power state transition
  • Pin induced
  • Hardware induced
  • Remote reset
  • Internal CPU event

persistent reboot reason bit

341-368
BitTypeReason
0Pinthermal pin `BP_THERMTRIP_L`이 trip됐습니다.
1Pinpower button을 4초 동안 눌렀습니다.
2Pinshutdown pin이 trip됐습니다.
4Remoteremote ASF power-off command를 받았습니다.
9Internalinternal CPU thermal limit에 도달했습니다.
16Pinsystem reset pin `BP_SYS_RST_L`이 trip됐습니다.
17Softwaresoftware가 PCI reset을 실행했습니다.
18Softwaresoftware가 reset-control register `0xCF9`에 `0x4`를 썼습니다.
19Softwaresoftware가 reset-control register `0xCF9`에 `0x6`을 썼습니다.
20Softwaresoftware가 reset-control register `0xCF9`에 `0xE`를 썼습니다.
21ACPI-stateACPI power-state transition이 발생했습니다.
22Pinkeyboard reset pin `KB_RST_L`이 trip됐습니다.
23Internalinternal CPU shutdown event가 발생했습니다.
24Hardwarefailed-boot timer 만료 전에 system boot가 끝나지 않았습니다.
25Hardwarehardware watchdog timer가 만료됐습니다.
26Remoteremote ASF reset command를 받았습니다.
27Internaluncorrected error가 data-fabric sync-flood event를 일으켰습니다.
29InternalFCH와 MP1의 warm-reset handshake가 실패했습니다.
30Internalparity error가 발생했습니다.
31Internalsoftware sync-flood event가 발생했습니다.

kernel은 boot 때 이 정보를 읽어 syslog에 출력합니다. random reboot가 발생했을 때 이 message는 다음으로 debugging할 component를 정하는 데 도움이 됩니다.