요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
1
==============================
2
Using the tracer for debugging
3
==============================
5
Copyright 2024 Google LLC.
7
:Author: Steven Rostedt <[email protected]>
8
:License: The GNU Free Documentation License, Version 1.2
9
(dual licensed under the GPL v2)
11
- Written for: 6.12
13
Introduction
14
------------
15
The tracing infrastructure can be very useful for debugging the Linux
16
kernel. This document is a place to add various methods of using the tracer
17
for debugging.
19
First, make sure that the tracefs file system is mounted::
21
$ sudo mount -t tracefs tracefs /sys/kernel/tracing
24
Using trace_printk()
25
--------------------
27
trace_printk() is a very lightweight utility that can be used in any context
28
inside the kernel, with the exception of "noinstr" sections. It can be used
29
in normal, softirq, interrupt and even NMI context. The trace data is
30
written to the tracing ring buffer in a lockless way. To make it even
31
lighter weight, when possible, it will only record the pointer to the format
32
string, and save the raw arguments into the buffer. The format and the
33
arguments will be post processed when the ring buffer is read. This way the
34
trace_printk() format conversions are not done during the hot path, where
35
the trace is being recorded.
37
trace_printk() is meant only for debugging, and should never be added into
38
a subsystem of the kernel. If you need debugging traces, add trace events
39
instead. If a trace_printk() is found in the kernel, the following will
40
appear in the dmesg::
42
**********************************************************
43
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
44
** **
45
** trace_printk() being used. Allocating extra memory. **
46
** **
47
** This means that this is a DEBUG kernel and it is **
48
** unsafe for production use. **
49
** **
50
** If you see this message and you are not debugging **
51
** the kernel, report this immediately to your vendor! **
52
** **
53
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
54
**********************************************************
56
Debugging kernel crashes
57
------------------------
58
There is various methods of acquiring the state of the system when a kernel
59
crash occurs. This could be from the oops message in printk, or one could
60
use kexec/kdump. But these just show what happened at the time of the crash.
61
It can be very useful in knowing what happened up to the point of the crash.
62
The tracing ring buffer, by default, is a circular buffer that will
63
overwrite older events with newer ones. When a crash happens, the content of
64
the ring buffer will be all the events that lead up to the crash.
66
There are several kernel command line parameters that can be used to help in
67
this. The first is "ftrace_dump_on_oops". This will dump the tracing ring
68
buffer when a oops occurs to the console. This can be useful if the console
69
is being logged somewhere. If a serial console is used, it may be prudent to
70
make sure the ring buffer is relatively small, otherwise the dumping of the
71
ring buffer may take several minutes to hours to finish. Here's an example
72
of the kernel command line::
74
ftrace_dump_on_oops trace_buf_size=50K
76
Note, the tracing buffer is made up of per CPU buffers where each of these
77
buffers is broken up into sub-buffers that are by default PAGE_SIZE. The
78
above trace_buf_size option above sets each of the per CPU buffers to 50K,
79
so, on a machine with 8 CPUs, that's actually 400K total.
81
Persistent buffers across boots
82
-------------------------------
83
If the system memory allows it, the tracing ring buffer can be specified at
84
a specific location in memory. If the location is the same across boots and
85
the memory is not modified, the tracing buffer can be retrieved from the
86
following boot. There's two ways to reserve memory for the use of the ring
87
buffer.
89
The more reliable way (on x86) is to reserve memory with the "memmap" kernel
90
command line option and then use that memory for the trace_instance. This
91
requires a bit of knowledge of the physical memory layout of the system. The
92
advantage of using this method, is that the memory for the ring buffer will
93
always be the same::
95
memmap==12M$0x284500000 trace_instance=boot_map@0x284500000:12M
97
The memmap above reserves 12 megabytes of memory at the physical memory
98
location 0x284500000. Then the trace_instance option will create a trace
99
instance "boot_map" at that same location with the same amount of memory
100
reserved. As the ring buffer is broke up into per CPU buffers, the 12
101
megabytes will be broken up evenly between those CPUs. If you have 8 CPUs,
102
each per CPU ring buffer will be 1.5 megabytes in size. Note, that also
103
includes meta data, so the amount of memory actually used by the ring buffer
104
will be slightly smaller.
106
Another more generic but less robust way to allocate a ring buffer mapping
107
at boot is with the "reserve_mem" option::
109
reserve_mem=12M:4096:trace trace_instance=boot_map@trace
111
The reserve_mem option above will find 12 megabytes that are available at
112
boot up, and align it by 4096 bytes. It will label this memory as "trace"
113
that can be used by later command line options.
115
The trace_instance option creates a "boot_map" instance and will use the
116
memory reserved by reserve_mem that was labeled as "trace". This method is
117
more generic but may not be as reliable. Due to KASLR, the memory reserved
118
by reserve_mem may not be located at the same location. If this happens,
119
then the ring buffer will not be from the previous boot and will be reset.
121
Sometimes, by using a larger alignment, it can keep KASLR from moving things
122
around in such a way that it will move the location of the reserve_mem. By
123
using a larger alignment, you may find better that the buffer is more
124
consistent to where it is placed::
126
reserve_mem=12M:0x2000000:trace trace_instance=boot_map@trace
128
On boot up, the memory reserved for the ring buffer is validated. It will go
129
through a series of tests to make sure that the ring buffer contains valid
130
data. If it is, it will then set it up to be available to read from the
131
instance. If it fails any of the tests, it will clear the entire ring buffer
132
and initialize it as new.
134
The layout of this mapped memory may not be consistent from kernel to
135
kernel, so only the same kernel is guaranteed to work if the mapping is
136
preserved. Switching to a different kernel version may find a different
137
layout and mark the buffer as invalid.
139
NB: Both the mapped address and size must be page aligned for the architecture.
141
Using trace_printk() in the boot instance
142
-----------------------------------------
143
By default, the content of trace_printk() goes into the top level tracing
144
instance. But this instance is never preserved across boots. To have the
145
trace_printk() content, and some other internal tracing go to the preserved
146
buffer (like dump stacks), either set the instance to be the trace_printk()
147
destination from the kernel command line, or set it after boot up via the
148
trace_printk_dest option.
150
After boot up::
152
echo 1 > /sys/kernel/tracing/instances/boot_map/options/trace_printk_dest
154
From the kernel command line::
156
reserve_mem=12M:4096:trace trace_instance=boot_map^traceprintk^traceoff@trace
158
If setting it from the kernel command line, it is recommended to also
159
disable tracing with the "traceoff" flag, and enable tracing after boot up.
160
Otherwise the trace from the most recent boot will be mixed with the trace
161
from the previous boot, and may make it confusing to read.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서 목적과 tracefs 준비
1-23이 문서는 Steven Rostedt가 Linux 6.12를 대상으로 작성한 tracer 디버깅 안내서다. 2024 Google LLC 저작물이며 GNU Free Documentation License 1.2와 GPL v2로 이중 사용 허가된다.
Linux kernel의 tracing infrastructure는 디버깅에 매우 유용하다. 이 문서는 tracer를 이용하는 여러 디버깅 방법을 모으는 장소다.
먼저 tracefs가 `/sys/kernel/tracing`에 mount되어 있는지 확인한다. 예시 명령은 `sudo mount -t tracefs tracefs /sys/kernel/tracing`이다.
Kernel tracing infrastructure→Mount tracefs
Mount tracefs→/sys/kernel/tracing
/sys/kernel/tracing→Tracing controls and buffers
디버깅 기능을 쓰기 전에 tracefs mount를 확인한다.
항목값
AuthorSteven Rostedt
Written forLinux 6.12
LicenseGNU FDL 1.2, dual licensed under GPL v2
원문에 명시된 작성 정보와 기준 버전이다.
==============================
Using the tracer for debugging
==============================
Copyright 2024 Google LLC.
:Author: Steven Rostedt <[email protected]>
:License: The GNU Free Documentation License, Version 1.2
(dual licensed under the GPL v2)
- Written for: 6.12
Introduction
------------
The tracing infrastructure can be very useful for debugging the Linux
kernel. This document is a place to add various methods of using the tracer
for debugging.
First, make sure that the tracefs file system is mounted::
$ sudo mount -t tracefs tracefs /sys/kernel/tracing
trace_printk()의 특성과 제한
24-55`trace_printk()`는 kernel 내부의 거의 모든 context에서 사용할 수 있는 매우 가벼운 utility다. `noinstr` section에서는 사용할 수 없지만 normal, softirq, interrupt, NMI context에서는 사용할 수 있다.
trace data는 lock 없이 tracing ring buffer에 기록된다. 가능한 경우 format string 자체 대신 그 pointer만 기록하고 raw argument를 buffer에 저장한다. ring buffer를 읽을 때 format과 argument를 후처리하므로 trace를 기록하는 hot path에서 format conversion을 하지 않는다.
`trace_printk()`는 디버깅 전용이며 kernel subsystem에 영구적으로 추가해서는 안 된다. 지속적인 debugging trace가 필요하면 trace event를 추가해야 한다.
kernel에서 `trace_printk()`가 발견되면 dmesg에 추가 메모리를 할당하며 production 용도로 안전하지 않은 DEBUG kernel이라는 큰 경고가 출력된다. 디버깅하지 않는데 이 메시지가 보이면 vendor에 즉시 보고해야 한다.
Context사용
Normal가능
softirq / interrupt가능
NMI가능
noinstr section불가
Production subsystem 코드추가 금지, trace event 사용
사용 가능한 execution context와 금지 영역을 구분한다.
trace_printk call→Format-string pointer
trace_printk call→Raw arguments
Pointer and arguments→Lockless ring buffer
Read ring buffer→Post-process formatting
hot path에서는 최소 정보만 저장하고 읽을 때 문자열을 완성한다.
Using trace_printk()
--------------------
trace_printk() is a very lightweight utility that can be used in any context
inside the kernel, with the exception of "noinstr" sections. It can be used
in normal, softirq, interrupt and even NMI context. The trace data is
written to the tracing ring buffer in a lockless way. To make it even
lighter weight, when possible, it will only record the pointer to the format
string, and save the raw arguments into the buffer. The format and the
arguments will be post processed when the ring buffer is read. This way the
trace_printk() format conversions are not done during the hot path, where
the trace is being recorded.
trace_printk() is meant only for debugging, and should never be added into
a subsystem of the kernel. If you need debugging traces, add trace events
instead. If a trace_printk() is found in the kernel, the following will
appear in the dmesg::
**********************************************************
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
** **
** trace_printk() being used. Allocating extra memory. **
** **
** This means that this is a DEBUG kernel and it is **
** unsafe for production use. **
** **
** If you see this message and you are not debugging **
** the kernel, report this immediately to your vendor! **
** **
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
**********************************************************
Kernel crash 직전 event 확보
56-80kernel crash 때 system 상태를 얻는 방법에는 printk의 oops message와 kexec/kdump가 있다. 그러나 이들은 crash 순간에 무슨 일이 있었는지만 보여 준다. crash까지 어떤 일이 이어졌는지 아는 것도 중요하다.
기본 tracing ring buffer는 오래된 event를 새 event로 덮어쓰는 circular buffer다. 따라서 crash가 발생하면 buffer에는 crash로 이어진 최신 event들이 남아 있다.
kernel command line의 `ftrace_dump_on_oops`는 oops 발생 시 tracing ring buffer를 console에 dump한다. console log를 저장하는 환경에서 유용하다.
serial console에서는 큰 ring buffer를 dump하는 데 수분에서 수시간이 걸릴 수 있으므로 buffer를 비교적 작게 두는 편이 좋다. 예시는 `ftrace_dump_on_oops trace_buf_size=50K`다.
tracing buffer는 CPU별 buffer로 구성되고 각 buffer는 기본적으로 PAGE_SIZE인 sub-buffer로 나뉜다. `trace_buf_size=50K`는 전체가 아니라 각 CPU buffer를 50K로 설정하므로 CPU가 8개면 총 400K다.
Continuous events→Per-CPU circular buffers
New event→Overwrite oldest event
Oops→ftrace_dump_on_oops
ftrace_dump_on_oops→Console log
circular ring buffer의 최신 event를 oops 때 console로 내보낸다.
CPU 수CPU별 크기총 크기
150K50K
850K400K
option은 CPU별 크기라는 점이 핵심이다.
Debugging kernel crashes
------------------------
There is various methods of acquiring the state of the system when a kernel
crash occurs. This could be from the oops message in printk, or one could
use kexec/kdump. But these just show what happened at the time of the crash.
It can be very useful in knowing what happened up to the point of the crash.
The tracing ring buffer, by default, is a circular buffer that will
overwrite older events with newer ones. When a crash happens, the content of
the ring buffer will be all the events that lead up to the crash.
There are several kernel command line parameters that can be used to help in
this. The first is "ftrace_dump_on_oops". This will dump the tracing ring
buffer when a oops occurs to the console. This can be useful if the console
is being logged somewhere. If a serial console is used, it may be prudent to
make sure the ring buffer is relatively small, otherwise the dumping of the
ring buffer may take several minutes to hours to finish. Here's an example
of the kernel command line::
ftrace_dump_on_oops trace_buf_size=50K
Note, the tracing buffer is made up of per CPU buffers where each of these
buffers is broken up into sub-buffers that are by default PAGE_SIZE. The
above trace_buf_size option above sets each of the per CPU buffers to 50K,
so, on a machine with 8 CPUs, that's actually 400K total.
부팅을 넘겨 보존하는 ring buffer
81-140system memory가 허용하면 tracing ring buffer를 특정 memory location에 둘 수 있다. 부팅 사이에 location이 같고 memory가 수정되지 않으면 다음 부팅에서 이전 tracing buffer를 회수할 수 있다. ring buffer용 memory를 예약하는 방법은 두 가지다.
x86에서 더 신뢰할 수 있는 방법은 kernel command line의 `memmap`으로 물리 메모리를 예약하고 그 위치를 `trace_instance`에 사용하는 것이다. physical memory layout을 알아야 하지만 ring buffer 위치가 항상 같다는 장점이 있다.
`memmap==12M$0x284500000 trace_instance=boot_map@0x284500000:12M`은 물리 주소 `0x284500000`에서 12 MiB를 예약하고 같은 위치와 크기로 `boot_map` instance를 만든다.
ring buffer는 CPU별로 균등 분할된다. CPU가 8개면 12 MiB 중 CPU별 1.5 MiB를 받는다. metadata도 이 공간에 들어가므로 실제 ring buffer data에 쓰는 양은 조금 더 작다.
더 일반적이지만 덜 견고한 방법은 `reserve_mem=12M:4096:trace trace_instance=boot_map@trace`다. 부팅 시 사용 가능한 12 MiB를 찾아 4096 byte에 맞춰 정렬하고 `trace` label을 붙인다. 뒤의 `trace_instance`가 이 label의 memory로 `boot_map`을 만든다.
KASLR 때문에 `reserve_mem`이 다음 부팅에서 같은 위치에 놓이지 않을 수 있다. 그러면 이전 부팅의 ring buffer로 인식되지 않아 초기화된다. `0x2000000`처럼 더 큰 alignment를 사용하면 위치가 더 일관될 수 있다.
부팅할 때 예약 ring buffer memory를 여러 test로 검증한다. 유효하면 instance에서 읽을 수 있게 구성하고, 하나라도 실패하면 ring buffer 전체를 지우고 새로 초기화한다.
mapped-memory layout은 kernel마다 같다고 보장되지 않으므로 보존 mapping은 같은 kernel에서만 작동한다고 보장된다. 다른 kernel version은 layout 차이로 buffer를 invalid로 판정할 수 있다. mapped address와 size는 모두 architecture page 경계에 맞아야 한다.
방법장점주의점
memmap + trace_instance부팅마다 같은 물리 주소, x86에서 신뢰성 높음physical memory layout 지식 필요
reserve_mem + trace_instancearchitecture에 덜 종속적KASLR로 위치가 달라질 수 있음
고정 물리 주소와 label 기반 동적 예약의 tradeoff다.
memmap 12M at 0x284500000→Reserved physical memory
trace_instance boot_map→Same address and size
12 MiB→Split across per-CPU buffers
고정 주소 예약과 trace instance가 같은 범위를 공유한다.
reserve_mem 12M aligned→Label trace
trace_instance boot_map@trace→Mapped ring buffer
Next boot validation→Reuse or reset
label로 예약 memory를 이후 option에 전달한다.
검증동작
모든 test 통과instance에서 이전 buffer 읽기 허용
하나라도 실패buffer 전체를 지우고 새로 초기화
다른 kernel layoutinvalid로 판단할 수 있음
보존 buffer를 읽을지 초기화할지 결정한다.
Persistent buffers across boots
-------------------------------
If the system memory allows it, the tracing ring buffer can be specified at
a specific location in memory. If the location is the same across boots and
the memory is not modified, the tracing buffer can be retrieved from the
following boot. There's two ways to reserve memory for the use of the ring
buffer.
The more reliable way (on x86) is to reserve memory with the "memmap" kernel
command line option and then use that memory for the trace_instance. This
requires a bit of knowledge of the physical memory layout of the system. The
advantage of using this method, is that the memory for the ring buffer will
always be the same::
memmap==12M$0x284500000 trace_instance=boot_map@0x284500000:12M
The memmap above reserves 12 megabytes of memory at the physical memory
location 0x284500000. Then the trace_instance option will create a trace
instance "boot_map" at that same location with the same amount of memory
reserved. As the ring buffer is broke up into per CPU buffers, the 12
megabytes will be broken up evenly between those CPUs. If you have 8 CPUs,
each per CPU ring buffer will be 1.5 megabytes in size. Note, that also
includes meta data, so the amount of memory actually used by the ring buffer
will be slightly smaller.
Another more generic but less robust way to allocate a ring buffer mapping
at boot is with the "reserve_mem" option::
reserve_mem=12M:4096:trace trace_instance=boot_map@trace
The reserve_mem option above will find 12 megabytes that are available at
boot up, and align it by 4096 bytes. It will label this memory as "trace"
that can be used by later command line options.
The trace_instance option creates a "boot_map" instance and will use the
memory reserved by reserve_mem that was labeled as "trace". This method is
more generic but may not be as reliable. Due to KASLR, the memory reserved
by reserve_mem may not be located at the same location. If this happens,
then the ring buffer will not be from the previous boot and will be reset.
Sometimes, by using a larger alignment, it can keep KASLR from moving things
around in such a way that it will move the location of the reserve_mem. By
using a larger alignment, you may find better that the buffer is more
consistent to where it is placed::
reserve_mem=12M:0x2000000:trace trace_instance=boot_map@trace
On boot up, the memory reserved for the ring buffer is validated. It will go
through a series of tests to make sure that the ring buffer contains valid
data. If it is, it will then set it up to be available to read from the
instance. If it fails any of the tests, it will clear the entire ring buffer
and initialize it as new.
The layout of this mapped memory may not be consistent from kernel to
kernel, so only the same kernel is guaranteed to work if the mapping is
preserved. Switching to a different kernel version may find a different
layout and mark the buffer as invalid.
NB: Both the mapped address and size must be page aligned for the architecture.
보존 instance로 trace_printk() 보내기
141-161기본적으로 `trace_printk()` 내용은 최상위 tracing instance로 들어가지만 이 instance는 부팅 사이에 보존되지 않는다.
`trace_printk()` 내용과 dump stack 같은 내부 trace를 보존 buffer에 넣으려면 kernel command line에서 그 instance를 destination으로 지정하거나, 부팅 뒤 `trace_printk_dest` option을 설정한다.
부팅 뒤에는 `echo 1 > /sys/kernel/tracing/instances/boot_map/options/trace_printk_dest`를 사용한다. command line에서는 `trace_instance=boot_map^traceprintk^traceoff@trace` flags를 사용한다.
command line에서 설정할 때는 `traceoff`로 tracing을 함께 비활성화하고 부팅 후 다시 활성화하는 것이 권장된다. 그렇지 않으면 최신 부팅의 trace가 이전 부팅 trace와 섞여 읽기 어려워질 수 있다.
trace_printk and internal traces→trace_printk_dest
trace_printk_dest→boot_map persistent instance
traceoff during boot→Enable after previous trace is read
top-level 대신 보존 instance를 명시적으로 목적지로 삼는다.
Using trace_printk() in the boot instance
-----------------------------------------
By default, the content of trace_printk() goes into the top level tracing
instance. But this instance is never preserved across boots. To have the
trace_printk() content, and some other internal tracing go to the preserved
buffer (like dump stacks), either set the instance to be the trace_printk()
destination from the kernel command line, or set it after boot up via the
trace_printk_dest option.
After boot up::
echo 1 > /sys/kernel/tracing/instances/boot_map/options/trace_printk_dest
From the kernel command line::
reserve_mem=12M:4096:trace trace_instance=boot_map^traceprintk^traceoff@trace
If setting it from the kernel command line, it is recommended to also
disable tracing with the "traceoff" flag, and enable tracing after boot up.
Otherwise the trace from the most recent boot will be mixed with the trace
from the previous boot, and may make it confusing to read.
요약·해설
debugging.rst:1-161trace_printk(), crash dump, memmap 또는 reserve_mem 기반 persistent ring buffer를 이용한 kernel tracer 디버깅 방법입니다.