요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
1
.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
2
.. [see the bottom of this file for redistribution information]
4
=========================================
5
How to verify bugs and bisect regressions
6
=========================================
8
This document describes how to check if some Linux kernel problem occurs in code
9
currently supported by developers -- to then explain how to locate the change
10
causing the issue, if it is a regression (e.g. did not happen with earlier
11
versions).
13
The text aims at people running kernels from mainstream Linux distributions on
14
commodity hardware who want to report a kernel bug to the upstream Linux
15
developers. Despite this intent, the instructions work just as well for users
16
who are already familiar with building their own kernels: they help avoid
17
mistakes occasionally made even by experienced developers.
19
..
20
Note: if you see this note, you are reading the text's source file. You
21
might want to switch to a rendered version: it makes it a lot easier to
22
read and navigate this document -- especially when you want to look something
23
up in the reference section, then jump back to where you left off.
24
..
25
Find the latest rendered version of this text here:
26
https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
28
The essence of the process (aka 'TL;DR')
29
========================================
31
*[If you are new to building or bisecting Linux, ignore this section and head
32
over to the* ':ref:`step-by-step guide <introguide_bissbs>`' *below. It utilizes
33
the same commands as this section while describing them in brief fashion. The
34
steps are nevertheless easy to follow and together with accompanying entries
35
in a reference section mention many alternatives, pitfalls, and additional
36
aspects, all of which might be essential in your present case.]*
38
**In case you want to check if a bug is present in code currently supported by
39
developers**, execute just the *preparations* and *segment 1*; while doing so,
40
consider the newest Linux kernel you regularly use to be the 'working' kernel.
41
In the following example that's assumed to be 6.0, which is why its sources
42
will be used to prepare the .config file.
44
**In case you face a regression**, follow the steps at least till the end of
45
*segment 2*. Then you can submit a preliminary report -- or continue with
46
*segment 3*, which describes how to perform a bisection needed for a
47
full-fledged regression report. In the following example 6.0.13 is assumed to be
48
the 'working' kernel and 6.1.5 to be the first 'broken', which is why 6.0
49
will be considered the 'good' release and used to prepare the .config file.
51
* **Preparations**: set up everything to build your own kernels::
53
# * Remove any software that depends on externally maintained kernel modules
54
# or builds any automatically during bootup.
55
# * Ensure Secure Boot permits booting self-compiled Linux kernels.
56
# * If you are not already running the 'working' kernel, reboot into it.
57
# * Install compilers and everything else needed for building Linux.
58
# * Ensure to have 15 Gigabyte free space in your home directory.
59
git clone -o mainline --no-checkout \
60
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
61
cd ~/linux/
62
git remote add -t master stable \
63
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
64
git switch --detach v6.0
65
# * Hint: if you used an existing clone, ensure no stale .config is around.
66
make olddefconfig
67
# * Ensure the former command picked the .config of the 'working' kernel.
68
# * Connect external hardware (USB keys, tokens, ...), start a VM, bring up
69
# VPNs, mount network shares, and briefly try the feature that is broken.
70
yes '' | make localmodconfig
71
./scripts/config --set-str CONFIG_LOCALVERSION '-local'
72
./scripts/config -e CONFIG_LOCALVERSION_AUTO
73
# * Note, when short on storage space, check the guide for an alternative:
74
./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
75
-e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
76
# * Hint: at this point you might want to adjust the build configuration;
77
# you'll have to, if you are running Debian.
78
make olddefconfig
79
cp .config ~/kernel-config-working
81
* **Segment 1**: build a kernel from the latest mainline codebase.
83
This among others checks if the problem was fixed already and which developers
84
later need to be told about the problem; in case of a regression, this rules
85
out a .config change as root of the problem.
87
a) Checking out latest mainline code::
89
cd ~/linux/
90
git switch --discard-changes --detach mainline/master
92
b) Build, install, and boot a kernel::
94
cp ~/kernel-config-working .config
95
make olddefconfig
96
make -j $(nproc --all)
97
# * Make sure there is enough disk space to hold another kernel:
98
df -h /boot/ /lib/modules/
99
# * Note: on Arch Linux, its derivatives and a few other distributions
100
# the following commands will do nothing at all or only part of the
101
# job. See the step-by-step guide for further details.
102
sudo make modules_install
103
command -v installkernel && sudo make install
104
# * Check how much space your self-built kernel actually needs, which
105
# enables you to make better estimates later:
106
du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
107
du -sh /lib/modules/$(make -s kernelrelease)/
108
# * Hint: the output of the following command will help you pick the
109
# right kernel from the boot menu:
110
make -s kernelrelease | tee -a ~/kernels-built
111
reboot
112
# * Once booted, ensure you are running the kernel you just built by
113
# checking if the output of the next two commands matches:
114
tail -n 1 ~/kernels-built
115
uname -r
116
cat /proc/sys/kernel/tainted
118
c) Check if the problem occurs with this kernel as well.
120
* **Segment 2**: ensure the 'good' kernel is also a 'working' kernel.
122
This among others verifies the trimmed .config file actually works well, as
123
bisecting with it otherwise would be a waste of time:
125
a) Start by checking out the sources of the 'good' version::
127
cd ~/linux/
128
git switch --discard-changes --detach v6.0
130
b) Build, install, and boot a kernel as described earlier in *segment 1,
131
section b* -- just feel free to skip the 'du' commands, as you have a rough
132
estimate already.
134
c) Ensure the feature that regressed with the 'broken' kernel actually works
135
with this one.
137
* **Segment 3**: perform and validate the bisection.
139
a) Retrieve the sources for your 'bad' version::
141
git remote set-branches --add stable linux-6.1.y
142
git fetch stable
144
b) Initialize the bisection::
146
cd ~/linux/
147
git bisect start
148
git bisect good v6.0
149
git bisect bad v6.1.5
151
c) Build, install, and boot a kernel as described earlier in *segment 1,
152
section b*.
154
In case building or booting the kernel fails for unrelated reasons, run
155
``git bisect skip``. In all other outcomes, check if the regressed feature
156
works with the newly built kernel. If it does, tell Git by executing
157
``git bisect good``; if it does not, run ``git bisect bad`` instead.
159
All three commands will make Git check out another commit; then re-execute
160
this step (e.g. build, install, boot, and test a kernel to then tell Git
161
the outcome). Do so again and again until Git shows which commit broke
162
things. If you run short of disk space during this process, check the
163
section 'Complementary tasks: cleanup during and after the process'
164
below.
166
d) Once your finished the bisection, put a few things away::
168
cd ~/linux/
169
git bisect log > ~/bisect-log
170
cp .config ~/bisection-config-culprit
171
git bisect reset
173
e) Try to verify the bisection result::
175
git switch --discard-changes --detach mainline/master
176
git revert --no-edit cafec0cacaca0
177
cp ~/kernel-config-working .config
178
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
180
This is optional, as some commits are impossible to revert. But if the
181
second command worked flawlessly, build, install, and boot one more kernel
182
kernel; just this time skip the first command copying the base .config file
183
over, as that already has been taken care off.
185
* **Complementary tasks**: cleanup during and after the process.
187
a) To avoid running out of disk space during a bisection, you might need to
188
remove some kernels you built earlier. You most likely want to keep those
189
you built during segment 1 and 2 around for a while, but you will most
190
likely no longer need kernels tested during the actual bisection
191
(Segment 3 c). You can list them in build order using::
193
ls -ltr /lib/modules/*-local*
195
To then for example erase a kernel that identifies itself as
196
'6.0-rc1-local-gcafec0cacaca0', use this::
198
sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
199
sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
200
# * Note, on some distributions kernel-install is missing
201
# or does only part of the job.
203
b) If you performed a bisection and successfully validated the result, feel
204
free to remove all kernels built during the actual bisection (Segment 3 c);
205
the kernels you built earlier and later you might want to keep around for
206
a week or two.
208
* **Optional task**: test a debug patch or a proposed fix later::
210
git fetch mainline
211
git switch --discard-changes --detach mainline/master
212
git apply /tmp/foobars-proposed-fix-v1.patch
213
cp ~/kernel-config-working .config
214
./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
216
Build, install, and boot a kernel as described in *segment 1, section b* --
217
but this time omit the first command copying the build configuration over,
218
as that has been taken care of already.
220
.. _introguide_bissbs:
222
Step-by-step guide on how to verify bugs and bisect regressions
223
===============================================================
225
This guide describes how to set up your own Linux kernels for investigating bugs
226
or regressions you intend to report. How far you want to follow the instructions
227
depends on your issue:
229
Execute all steps till the end of *segment 1* to **verify if your kernel problem
230
is present in code supported by Linux kernel developers**. If it is, you are all
231
set to report the bug -- unless it did not happen with earlier kernel versions,
232
as then your want to at least continue with *segment 2* to **check if the issue
233
qualifies as regression** which receive priority treatment. Depending on the
234
outcome you then are ready to report a bug or submit a preliminary regression
235
report; instead of the latter your could also head straight on and follow
236
*segment 3* to **perform a bisection** for a full-fledged regression report
237
developers are obliged to act upon.
239
:ref:`Preparations: set up everything to build your own kernels <introprep_bissbs>`.
241
:ref:`Segment 1: try to reproduce the problem with the latest codebase <introlatestcheck_bissbs>`.
243
:ref:`Segment 2: check if the kernels you build work fine <introworkingcheck_bissbs>`.
245
:ref:`Segment 3: perform a bisection and validate the result <introbisect_bissbs>`.
247
:ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.
249
:ref:`Optional tasks: test reverts, patches, or later versions <introoptional_bissbs>`.
251
The steps in each segment illustrate the important aspects of the process, while
252
a comprehensive reference section holds additional details for almost all of the
253
steps. The reference section sometimes also outlines alternative approaches,
254
pitfalls, as well as problems that might occur at the particular step -- and how
255
to get things rolling again.
257
For further details on how to report Linux kernel issues or regressions check
258
out Documentation/admin-guide/reporting-issues.rst, which works in conjunction
259
with this document. It among others explains why you need to verify bugs with
260
the latest 'mainline' kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6),
261
even if you face a problem with a kernel from a 'stable/longterm' series
262
(say 6.0.13).
264
For users facing a regression that document also explains why sending a
265
preliminary report after segment 2 might be wise, as the regression and its
266
culprit might be known already. For further details on what actually qualifies
267
as a regression check out Documentation/admin-guide/reporting-regressions.rst.
269
If you run into any problems while following this guide or have ideas how to
270
improve it, :ref:`please let the kernel developers know <submit_improvements_vbbr>`.
272
.. _introprep_bissbs:
274
Preparations: set up everything to build your own kernels
275
---------------------------------------------------------
277
The following steps lay the groundwork for all further tasks.
279
Note: the instructions assume you are building and testing on the same
280
machine; if you want to compile the kernel on another system, check
281
:ref:`Build kernels on a different machine <buildhost_bis>` below.
283
.. _backup_bissbs:
285
* Create a fresh backup and put system repair and restore tools at hand, just
286
to be prepared for the unlikely case of something going sideways.
288
[:ref:`details <backup_bisref>`]
290
.. _vanilla_bissbs:
292
* Remove all software that depends on externally developed kernel drivers or
293
builds them automatically. That includes but is not limited to DKMS, openZFS,
294
VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module).
296
[:ref:`details <vanilla_bisref>`]
298
.. _secureboot_bissbs:
300
* On platforms with 'Secure Boot' or similar solutions, prepare everything to
301
ensure the system will permit your self-compiled kernel to boot. The
302
quickest and easiest way to achieve this on commodity x86 systems is to
303
disable such techniques in the BIOS setup utility; alternatively, remove
304
their restrictions through a process initiated by
305
``mokutil --disable-validation``.
307
[:ref:`details <secureboot_bisref>`]
309
.. _rangecheck_bissbs:
311
* Determine the kernel versions considered 'good' and 'bad' throughout this
312
guide:
314
* Do you follow this guide to verify if a bug is present in the code the
315
primary developers care for? Then consider the version of the newest kernel
316
you regularly use currently as 'good' (e.g. 6.0, 6.0.13, or 6.1-rc2).
318
* Do you face a regression, e.g. something broke or works worse after
319
switching to a newer kernel version? In that case it depends on the version
320
range during which the problem appeared:
322
* Something regressed when updating from a stable/longterm release
323
(say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a
324
stable/longterm version based on one (say 6.1.5)? Then consider the
325
mainline release your working kernel is based on to be the 'good'
326
version (e.g. 6.0) and the first version to be broken as the 'bad' one
327
(e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assumed
328
that 6.0 is fine; this hypothesis will be checked in segment 2.
330
* Something regressed when switching from one mainline version (say 6.0) to
331
a later one (like 6.1-rc1) or a stable/longterm release based on it
332
(say 6.1.5)? Then regard the last working version (e.g. 6.0) as 'good' and
333
the first broken (e.g. 6.1-rc1 or 6.1.5) as 'bad'.
335
* Something regressed when updating within a stable/longterm series (say
336
from 6.0.13 to 6.0.15)? Then consider those versions as 'good' and 'bad'
337
(e.g. 6.0.13 and 6.0.15), as you need to bisect within that series.
339
*Note, do not confuse 'good' version with 'working' kernel; the latter term
340
throughout this guide will refer to the last kernel that has been working
341
fine.*
343
[:ref:`details <rangecheck_bisref>`]
345
.. _bootworking_bissbs:
347
* Boot into the 'working' kernel and briefly use the apparently broken feature.
349
[:ref:`details <bootworking_bisref>`]
351
.. _diskspace_bissbs:
353
* Ensure to have enough free space for building Linux. 15 Gigabyte in your home
354
directory should typically suffice. If you have less available, be sure to pay
355
attention to later steps about retrieving the Linux sources and handling of
356
debug symbols: both explain approaches reducing the amount of space, which
357
should allow you to master these tasks with about 4 Gigabytes free space.
359
[:ref:`details <diskspace_bisref>`]
361
.. _buildrequires_bissbs:
363
* Install all software required to build a Linux kernel. Often you will need:
364
'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl',
365
'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The
366
reference section shows how to quickly install those on various popular Linux
367
distributions.
369
[:ref:`details <buildrequires_bisref>`]
371
.. _sources_bissbs:
373
* Retrieve the mainline Linux sources; then change into the directory holding
374
them, as all further commands in this guide are meant to be executed from
375
there.
377
*Note, the following describe how to retrieve the sources using a full
378
mainline clone, which downloads about 2,75 GByte as of early 2024. The*
379
:ref:`reference section describes two alternatives <sources_bisref>` *:
380
one downloads less than 500 MByte, the other works better with unreliable
381
internet connections.*
383
Execute the following command to retrieve a fresh mainline codebase while
384
preparing things to add branches for stable/longterm series later::
386
git clone -o mainline --no-checkout \
387
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
388
cd ~/linux/
389
git remote add -t master stable \
390
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
392
[:ref:`details <sources_bisref>`]
394
.. _stablesources_bissbs:
396
* Is one of the versions you earlier established as 'good' or 'bad' a stable or
397
longterm release (say 6.1.5)? Then download the code for the series it belongs
398
to ('linux-6.1.y' in this example)::
400
git remote set-branches --add stable linux-6.1.y
401
git fetch stable
403
.. _oldconfig_bissbs:
405
* Start preparing a kernel build configuration (the '.config' file).
407
Before doing so, ensure you are still running the 'working' kernel an earlier
408
step told you to boot; if you are unsure, check the current kernelrelease
409
identifier using ``uname -r``.
411
Afterwards check out the source code for the version earlier established as
412
'good'. In the following example command this is assumed to be 6.0; note that
413
the version number in this and all later Git commands needs to be prefixed
414
with a 'v'::
416
git switch --discard-changes --detach v6.0
418
Now create a build configuration file::
420
make olddefconfig
422
The kernel build scripts then will try to locate the build configuration file
423
for the running kernel and then adjust it for the needs of the kernel sources
424
you checked out. While doing so, it will print a few lines you need to check.
426
Look out for a line starting with '# using defaults found in'. It should be
427
followed by a path to a file in '/boot/' that contains the release identifier
428
of your currently working kernel. If the line instead continues with something
429
like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find
430
the .config file for your running kernel -- in which case you have to put one
431
there manually, as explained in the reference section.
433
In case you can not find such a line, look for one containing '# configuration
434
written to .config'. If that's the case you have a stale build configuration
435
lying around. Unless you intend to use it, delete it; afterwards run
436
'make olddefconfig' again and check if it now picked up the right config file
437
as base.
439
[:ref:`details <oldconfig_bisref>`]
441
.. _localmodconfig_bissbs:
443
* Disable any kernel modules apparently superfluous for your setup. This is
444
optional, but especially wise for bisections, as it speeds up the build
445
process enormously -- at least unless the .config file picked up in the
446
previous step was already tailored to your and your hardware needs, in which
447
case you should skip this step.
449
To prepare the trimming, connect external hardware you occasionally use (USB
450
keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted
451
since you started that guide, ensure that you tried using the feature causing
452
trouble since you started the system. Only then trim your .config::
454
yes '' | make localmodconfig
456
There is a catch to this, as the 'apparently' in initial sentence of this step
457
and the preparation instructions already hinted at:
459
The 'localmodconfig' target easily disables kernel modules for features only
460
used occasionally -- like modules for external peripherals not yet connected
461
since booting, virtualization software not yet utilized, VPN tunnels, and a
462
few other things. That's because some tasks rely on kernel modules Linux only
463
loads when you execute tasks like the aforementioned ones for the first time.
465
This drawback of localmodconfig is nothing you should lose sleep over, but
466
something to keep in mind: if something is misbehaving with the kernels built
467
during this guide, this is most likely the reason. You can reduce or nearly
468
eliminate the risk with tricks outlined in the reference section; but when
469
building a kernel just for quick testing purposes this is usually not worth
470
spending much effort on, as long as it boots and allows to properly test the
471
feature that causes trouble.
473
[:ref:`details <localmodconfig_bisref>`]
475
.. _tagging_bissbs:
477
* Ensure all the kernels you will build are clearly identifiable using a special
478
tag and a unique version number::
480
./scripts/config --set-str CONFIG_LOCALVERSION '-local'
481
./scripts/config -e CONFIG_LOCALVERSION_AUTO
483
[:ref:`details <tagging_bisref>`]
485
.. _debugsymbols_bissbs:
487
* Decide how to handle debug symbols.
489
In the context of this document it is often wise to enable them, as there is a
490
decent chance you will need to decode a stack trace from a 'panic', 'Oops',
491
'warning', or 'BUG'::
493
./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
494
-e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
496
But if you are extremely short on storage space, you might want to disable
497
debug symbols instead::
499
./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
500
-d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
502
[:ref:`details <debugsymbols_bisref>`]
504
.. _configmods_bissbs:
506
* Check if you may want or need to adjust some other kernel configuration
507
options:
509
* Are you running Debian? Then you want to avoid known problems by performing
510
additional adjustments explained in the reference section.
512
[:ref:`details <configmods_distros_bisref>`].
514
* If you want to influence other aspects of the configuration, do so now using
515
your preferred tool. Note, to use make targets like 'menuconfig' or
516
'nconfig', you will need to install the development files of ncurses; for
517
'xconfig' you likewise need the Qt5 or Qt6 headers.
519
[:ref:`details <configmods_individual_bisref>`].
521
.. _saveconfig_bissbs:
523
* Reprocess the .config after the latest adjustments and store it in a safe
524
place::
526
make olddefconfig
527
cp .config ~/kernel-config-working
529
[:ref:`details <saveconfig_bisref>`]
531
.. _introlatestcheck_bissbs:
533
Segment 1: try to reproduce the problem with the latest codebase
534
----------------------------------------------------------------
536
The following steps verify if the problem occurs with the code currently
537
supported by developers. In case you face a regression, it also checks that the
538
problem is not caused by some .config change, as reporting the issue then would
539
be a waste of time. [:ref:`details <introlatestcheck_bisref>`]
541
.. _checkoutmaster_bissbs:
543
* Check out the latest Linux codebase.
545
* Are your 'good' and 'bad' versions from the same stable or longterm series?
546
Then check the `front page of kernel.org <https://kernel.org/>`_: if it
547
lists a release from that series without an '[EOL]' tag, checkout the series
548
latest version ('linux-6.1.y' in the following example)::
550
cd ~/linux/
551
git switch --discard-changes --detach stable/linux-6.1.y
553
Your series is unsupported, if is not listed or carrying a 'end of life'
554
tag. In that case you might want to check if a successor series (say
555
linux-6.2.y) or mainline (see next point) fix the bug.
557
* In all other cases, run::
559
cd ~/linux/
560
git switch --discard-changes --detach mainline/master
562
[:ref:`details <checkoutmaster_bisref>`]
564
.. _build_bissbs:
566
* Build the image and the modules of your first kernel using the config file you
567
prepared::
569
cp ~/kernel-config-working .config
570
make olddefconfig
571
make -j $(nproc --all)
573
If you want your kernel packaged up as deb, rpm, or tar file, see the
574
reference section for alternatives, which obviously will require other
575
steps to install as well.
577
[:ref:`details <build_bisref>`]
579
.. _install_bissbs:
581
* Install your newly built kernel.
583
Before doing so, consider checking if there is still enough space for it::
585
df -h /boot/ /lib/modules/
587
For now assume 150 MByte in /boot/ and 200 in /lib/modules/ will suffice; how
588
much your kernels actually require will be determined later during this guide.
590
Now install the kernel's modules and its image, which will be stored in
591
parallel to the your Linux distribution's kernels::
593
sudo make modules_install
594
command -v installkernel && sudo make install
596
The second command ideally will take care of three steps required at this
597
point: copying the kernel's image to /boot/, generating an initramfs, and
598
adding an entry for both to the boot loader's configuration.
600
Sadly some distributions (among them Arch Linux, its derivatives, and many
601
immutable Linux distributions) will perform none or only some of those tasks.
602
You therefore want to check if all of them were taken care of and manually
603
perform those that were not. The reference section provides further details on
604
that; your distribution's documentation might help, too.
606
Once you figured out the steps needed at this point, consider writing them
607
down: if you will build more kernels as described in segment 2 and 3, you will
608
have to perform those again after executing ``command -v installkernel [...]``.
610
[:ref:`details <install_bisref>`]
612
.. _storagespace_bissbs:
614
* In case you plan to follow this guide further, check how much storage space
615
the kernel, its modules, and other related files like the initramfs consume::
617
du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
618
du -sh /lib/modules/$(make -s kernelrelease)/
620
Write down or remember those two values for later: they enable you to prevent
621
running out of disk space accidentally during a bisection.
623
[:ref:`details <storagespace_bisref>`]
625
.. _kernelrelease_bissbs:
627
* Show and store the kernelrelease identifier of the kernel you just built::
629
make -s kernelrelease | tee -a ~/kernels-built
631
Remember the identifier momentarily, as it will help you pick the right kernel
632
from the boot menu upon restarting.
634
* Reboot into your newly built kernel. To ensure your actually started the one
635
you just built, you might want to verify if the output of these commands
636
matches::
638
tail -n 1 ~/kernels-built
639
uname -r
641
.. _tainted_bissbs:
643
* Check if the kernel marked itself as 'tainted'::
645
cat /proc/sys/kernel/tainted
647
If that command does not return '0', check the reference section, as the cause
648
for this might interfere with your testing.
650
[:ref:`details <tainted_bisref>`]
652
.. _recheckbroken_bissbs:
654
* Verify if your bug occurs with the newly built kernel. If it does not, check
655
out the instructions in the reference section to ensure nothing went sideways
656
during your tests.
658
[:ref:`details <recheckbroken_bisref>`]
660
.. _recheckstablebroken_bissbs:
662
* Did you just built a stable or longterm kernel? And were you able to reproduce
663
the regression with it? Then you should test the latest mainline codebase as
664
well, because the result determines which developers the bug must be submitted
665
to.
667
To prepare that test, check out current mainline::
669
cd ~/linux/
670
git switch --discard-changes --detach mainline/master
672
Now use the checked out code to build and install another kernel using the
673
commands the earlier steps already described in more detail::
675
cp ~/kernel-config-working .config
676
make olddefconfig
677
make -j $(nproc --all)
678
# * Check if the free space suffices holding another kernel:
679
df -h /boot/ /lib/modules/
680
sudo make modules_install
681
command -v installkernel && sudo make install
682
make -s kernelrelease | tee -a ~/kernels-built
683
reboot
685
Confirm you booted the kernel you intended to start and check its tainted
686
status::
688
tail -n 1 ~/kernels-built
689
uname -r
690
cat /proc/sys/kernel/tainted
692
Now verify if this kernel is showing the problem. If it does, then you need
693
to report the bug to the primary developers; if it does not, report it to the
694
stable team. See Documentation/admin-guide/reporting-issues.rst for details.
696
[:ref:`details <recheckstablebroken_bisref>`]
698
Do you follow this guide to verify if a problem is present in the code
699
currently supported by Linux kernel developers? Then you are done at this
700
point. If you later want to remove the kernel you just built, check out
701
:ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.
703
In case you face a regression, move on and execute at least the next segment
704
as well.
706
.. _introworkingcheck_bissbs:
708
Segment 2: check if the kernels you build work fine
709
---------------------------------------------------
711
In case of a regression, you now want to ensure the trimmed configuration file
712
you created earlier works as expected; a bisection with the .config file
713
otherwise would be a waste of time. [:ref:`details <introworkingcheck_bisref>`]
715
.. _recheckworking_bissbs:
717
* Build your own variant of the 'working' kernel and check if the feature that
718
regressed works as expected with it.
720
Start by checking out the sources for the version earlier established as
721
'good' (once again assumed to be 6.0 here)::
723
cd ~/linux/
724
git switch --discard-changes --detach v6.0
726
Now use the checked out code to configure, build, and install another kernel
727
using the commands the previous subsection explained in more detail::
729
cp ~/kernel-config-working .config
730
make olddefconfig
731
make -j $(nproc --all)
732
# * Check if the free space suffices holding another kernel:
733
df -h /boot/ /lib/modules/
734
sudo make modules_install
735
command -v installkernel && sudo make install
736
make -s kernelrelease | tee -a ~/kernels-built
737
reboot
739
When the system booted, you may want to verify once again that the
740
kernel you started is the one you just built::
742
tail -n 1 ~/kernels-built
743
uname -r
745
Now check if this kernel works as expected; if not, consult the reference
746
section for further instructions.
748
[:ref:`details <recheckworking_bisref>`]
750
.. _introbisect_bissbs:
752
Segment 3: perform the bisection and validate the result
753
--------------------------------------------------------
755
With all the preparations and precaution builds taken care of, you are now ready
756
to begin the bisection. This will make you build quite a few kernels -- usually
757
about 15 in case you encountered a regression when updating to a newer series
758
(say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build
759
configuration created earlier this works a lot faster than many people assume:
760
overall on average it will often just take about 10 to 15 minutes to compile
761
each kernel on commodity x86 machines.
763
.. _bisectstart_bissbs:
765
* Start the bisection and tell Git about the versions earlier established as
766
'good' (6.0 in the following example command) and 'bad' (6.1.5)::
768
cd ~/linux/
769
git bisect start
770
git bisect good v6.0
771
git bisect bad v6.1.5
773
[:ref:`details <bisectstart_bisref>`]
775
.. _bisectbuild_bissbs:
777
* Now use the code Git checked out to build, install, and boot a kernel using
778
the commands introduced earlier::
780
cp ~/kernel-config-working .config
781
make olddefconfig
782
make -j $(nproc --all)
783
# * Check if the free space suffices holding another kernel:
784
df -h /boot/ /lib/modules/
785
sudo make modules_install
786
command -v installkernel && sudo make install
787
make -s kernelrelease | tee -a ~/kernels-built
788
reboot
790
If compilation fails for some reason, run ``git bisect skip`` and restart
791
executing the stack of commands from the beginning.
793
In case you skipped the 'test latest codebase' step in the guide, check its
794
description as for why the 'df [...]' and 'make -s kernelrelease [...]'
795
commands are here.
797
Important note: the latter command from this point on will print release
798
identifiers that might look odd or wrong to you -- which they are not, as it's
799
totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0'
800
if you bisect between versions 6.1 and 6.2 for example.
802
[:ref:`details <bisectbuild_bisref>`]
804
.. _bisecttest_bissbs:
806
* Now check if the feature that regressed works in the kernel you just built.
808
You again might want to start by making sure the kernel you booted is the one
809
you just built::
811
cd ~/linux/
812
tail -n 1 ~/kernels-built
813
uname -r
815
Now verify if the feature that regressed works at this kernel bisection point.
816
If it does, run this::
818
git bisect good
820
If it does not, run this::
822
git bisect bad
824
Be sure about what you tell Git, as getting this wrong just once will send the
825
rest of the bisection totally off course.
827
While the bisection is ongoing, Git will use the information you provided to
828
find and check out another bisection point for you to test. While doing so, it
829
will print something like 'Bisecting: 675 revisions left to test after this
830
(roughly 10 steps)' to indicate how many further changes it expects to be
831
tested. Now build and install another kernel using the instructions from the
832
previous step; afterwards follow the instructions in this step again.
834
Repeat this again and again until you finish the bisection -- that's the case
835
when Git after tagging a change as 'good' or 'bad' prints something like
836
'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right
837
afterwards it will show some details about the culprit including the patch
838
description of the change. The latter might fill your terminal screen, so you
839
might need to scroll up to see the message mentioning the culprit;
840
alternatively, run ``git bisect log > ~/bisection-log``.
842
[:ref:`details <bisecttest_bisref>`]
844
.. _bisectlog_bissbs:
846
* Store Git's bisection log and the current .config file in a safe place before
847
telling Git to reset the sources to the state before the bisection::
849
cd ~/linux/
850
git bisect log > ~/bisection-log
851
cp .config ~/bisection-config-culprit
852
git bisect reset
854
[:ref:`details <bisectlog_bisref>`]
856
.. _revert_bissbs:
858
* Try reverting the culprit on top of latest mainline to see if this fixes your
859
regression.
861
This is optional, as it might be impossible or hard to realize. The former is
862
the case, if the bisection determined a merge commit as the culprit; the
863
latter happens if other changes depend on the culprit. But if the revert
864
succeeds, it is worth building another kernel, as it validates the result of
865
a bisection, which can easily deroute; it furthermore will let kernel
866
developers know, if they can resolve the regression with a quick revert.
868
Begin by checking out the latest codebase depending on the range you bisected:
870
* Did you face a regression within a stable/longterm series (say between
871
6.0.13 and 6.0.15) that does not happen in mainline? Then check out the
872
latest codebase for the affected series like this::
874
git fetch stable
875
git switch --discard-changes --detach linux-6.0.y
877
* In all other cases check out latest mainline::
879
git fetch mainline
880
git switch --discard-changes --detach mainline/master
882
If you bisected a regression within a stable/longterm series that also
883
happens in mainline, there is one more thing to do: look up the mainline
884
commit-id. To do so, use a command like ``git show abcdcafecabcd`` to
885
view the patch description of the culprit. There will be a line near
886
the top which looks like 'commit cafec0cacaca0 upstream.' or
887
'Upstream commit cafec0cacaca0'; use that commit-id in the next command
888
and not the one the bisection blamed.
890
Now try reverting the culprit by specifying its commit id::
892
git revert --no-edit cafec0cacaca0
894
If that fails, give up trying and move on to the next step; if it works,
895
adjust the tag to facilitate the identification and prevent accidentally
896
overwriting another kernel::
898
cp ~/kernel-config-working .config
899
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
901
Build a kernel using the familiar command sequence, just without copying the
902
the base .config over::
904
make olddefconfig &&
905
make -j $(nproc --all)
906
# * Check if the free space suffices holding another kernel:
907
df -h /boot/ /lib/modules/
908
sudo make modules_install
909
command -v installkernel && sudo make install
910
make -s kernelrelease | tee -a ~/kernels-built
911
reboot
913
Now check one last time if the feature that made you perform a bisection works
914
with that kernel: if everything went well, it should not show the regression.
916
[:ref:`details <revert_bisref>`]
918
.. _introclosure_bissbs:
920
Complementary tasks: cleanup during and after the bisection
921
-----------------------------------------------------------
923
During and after following this guide you might want or need to remove some of
924
the kernels you installed: the boot menu otherwise will become confusing or
925
space might run out.
927
.. _makeroom_bissbs:
929
* To remove one of the kernels you installed, look up its 'kernelrelease'
930
identifier. This guide stores them in '~/kernels-built', but the following
931
command will print them as well::
933
ls -ltr /lib/modules/*-local*
935
You in most situations want to remove the oldest kernels built during the
936
actual bisection (e.g. segment 3 of this guide). The two ones you created
937
beforehand (e.g. to test the latest codebase and the version considered
938
'good') might become handy to verify something later -- thus better keep them
939
around, unless you are really short on storage space.
941
To remove the modules of a kernel with the kernelrelease identifier
942
'*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its
943
modules::
945
sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
947
Afterwards try the following command::
949
sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
951
On quite a few distributions this will delete all other kernel files installed
952
while also removing the kernel's entry from the boot menu. But on some
953
distributions kernel-install does not exist or leaves boot-loader entries or
954
kernel image and related files behind; in that case remove them as described
955
in the reference section.
957
[:ref:`details <makeroom_bisref>`]
959
.. _finishingtouch_bissbs:
961
* Once you have finished the bisection, do not immediately remove anything you
962
set up, as you might need a few things again. What is safe to remove depends
963
on the outcome of the bisection:
965
* Could you initially reproduce the regression with the latest codebase and
966
after the bisection were able to fix the problem by reverting the culprit on
967
top of the latest codebase? Then you want to keep those two kernels around
968
for a while, but safely remove all others with a '-local' in the release
969
identifier.
971
* Did the bisection end on a merge-commit or seems questionable for other
972
reasons? Then you want to keep as many kernels as possible around for a few
973
days: it's pretty likely that you will be asked to recheck something.
975
* In other cases it likely is a good idea to keep the following kernels around
976
for some time: the one built from the latest codebase, the one created from
977
the version considered 'good', and the last three or four you compiled
978
during the actual bisection process.
980
[:ref:`details <finishingtouch_bisref>`]
982
.. _introoptional_bissbs:
984
Optional: test reverts, patches, or later versions
985
--------------------------------------------------
987
While or after reporting a bug, you might want or potentially will be asked to
988
test reverts, debug patches, proposed fixes, or other versions. In that case
989
follow these instructions.
991
* Update your Git clone and check out the latest code.
993
* In case you want to test mainline, fetch its latest changes before checking
994
its code out::
996
git fetch mainline
997
git switch --discard-changes --detach mainline/master
999
* In case you want to test a stable or longterm kernel, first add the branch
1000
holding the series you are interested in (6.2 in the example), unless you
1001
already did so earlier::
1003
git remote set-branches --add stable linux-6.2.y
1005
Then fetch the latest changes and check out the latest version from the
1006
series::
1008
git fetch stable
1009
git switch --discard-changes --detach stable/linux-6.2.y
1011
* Copy your kernel build configuration over::
1013
cp ~/kernel-config-working .config
1015
* Your next step depends on what you want to do:
1017
* In case you just want to test the latest codebase, head to the next step,
1018
you are already all set.
1020
* In case you want to test if a revert fixes an issue, revert one or multiple
1021
changes by specifying their commit ids::
1023
git revert --no-edit cafec0cacaca0
1025
Now give that kernel a special tag to facilitates its identification and
1026
prevent accidentally overwriting another kernel::
1028
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
1030
* In case you want to test a patch, store the patch in a file like
1031
'/tmp/foobars-proposed-fix-v1.patch' and apply it like this::
1033
git apply /tmp/foobars-proposed-fix-v1.patch
1035
In case of multiple patches, repeat this step with the others.
1037
Now give that kernel a special tag to facilitates its identification and
1038
prevent accidentally overwriting another kernel::
1040
./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
1042
* Build a kernel using the familiar commands, just without copying the kernel
1043
build configuration over, as that has been taken care of already::
1045
make olddefconfig &&
1046
make -j $(nproc --all)
1047
# * Check if the free space suffices holding another kernel:
1048
df -h /boot/ /lib/modules/
1049
sudo make modules_install
1050
command -v installkernel && sudo make install
1051
make -s kernelrelease | tee -a ~/kernels-built
1052
reboot
1054
* Now verify you booted the newly built kernel and check it.
1056
[:ref:`details <introoptional_bisref>`]
1058
.. _submit_improvements_vbbr:
1060
Conclusion
1061
----------
1063
You have reached the end of the step-by-step guide.
1065
Did you run into trouble following any of the above steps not cleared up by the
1066
reference section below? Did you spot errors? Or do you have ideas how to
1067
improve the guide?
1069
If any of that applies, please take a moment and let the maintainer of this
1070
document know by email (Thorsten Leemhuis <[email protected]>), ideally while
1071
CCing the Linux docs mailing list ([email protected]). Such feedback is
1072
vital to improve this text further, which is in everybody's interest, as it
1073
will enable more people to master the task described here -- and hopefully also
1074
improve similar guides inspired by this one.
1077
Reference section for the step-by-step guide
1078
============================================
1080
This section holds additional information for almost all the items in the above
1081
step-by-step guide.
1083
Preparations for building your own kernels
1084
------------------------------------------
1086
*The steps in this section lay the groundwork for all further tests.*
1087
[:ref:`... <introprep_bissbs>`]
1089
The steps in all later sections of this guide depend on those described here.
1091
[:ref:`back to step-by-step guide <introprep_bissbs>`].
1093
.. _backup_bisref:
1095
Prepare for emergencies
1096
~~~~~~~~~~~~~~~~~~~~~~~
1098
*Create a fresh backup and put system repair and restore tools at hand.*
1099
[:ref:`... <backup_bissbs>`]
1101
Remember, you are dealing with computers, which sometimes do unexpected things
1102
-- especially if you fiddle with crucial parts like the kernel of an operating
1103
system. That's what you are about to do in this process. Hence, better prepare
1104
for something going sideways, even if that should not happen.
1106
[:ref:`back to step-by-step guide <backup_bissbs>`]
1108
.. _vanilla_bisref:
1110
Remove anything related to externally maintained kernel modules
1111
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1113
*Remove all software that depends on externally developed kernel drivers or
1114
builds them automatically.* [:ref:`...<vanilla_bissbs>`]
1116
Externally developed kernel modules can easily cause trouble during a bisection.
1118
But there is a more important reason why this guide contains this step: most
1119
kernel developers will not care about reports about regressions occurring with
1120
kernels that utilize such modules. That's because such kernels are not
1121
considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst
1122
explains in more detail.
1124
[:ref:`back to step-by-step guide <vanilla_bissbs>`]
1126
.. _secureboot_bisref:
1128
Deal with techniques like Secure Boot
1129
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1131
*On platforms with 'Secure Boot' or similar techniques, prepare everything to
1132
ensure the system will permit your self-compiled kernel to boot later.*
1133
[:ref:`... <secureboot_bissbs>`]
1135
Many modern systems allow only certain operating systems to start; that's why
1136
they reject booting self-compiled kernels by default.
1138
You ideally deal with this by making your platform trust your self-built kernels
1139
with the help of a certificate. How to do that is not described
1140
here, as it requires various steps that would take the text too far away from
1141
its purpose; 'Documentation/admin-guide/module-signing.rst' and various web
1142
sides already explain everything needed in more detail.
1144
Temporarily disabling solutions like Secure Boot is another way to make your own
1145
Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup
1146
utility; the required steps vary a lot between machines and therefore cannot be
1147
described here.
1149
On mainstream x86 Linux distributions there is a third and universal option:
1150
disable all Secure Boot restrictions for your Linux environment. You can
1151
initiate this process by running ``mokutil --disable-validation``; this will
1152
tell you to create a one-time password, which is safe to write down. Now
1153
restart; right after your BIOS performed all self-tests the bootloader Shim will
1154
show a blue box with a message 'Press any key to perform MOK management'. Hit
1155
some key before the countdown exposes, which will open a menu. Choose 'Change
1156
Secure Boot state'. Shim's 'MokManager' will now ask you to enter three
1157
randomly chosen characters from the one-time password specified earlier. Once
1158
you provided them, confirm you really want to disable the validation.
1159
Afterwards, permit MokManager to reboot the machine.
1161
[:ref:`back to step-by-step guide <secureboot_bissbs>`]
1163
.. _bootworking_bisref:
1165
Boot the last kernel that was working
1166
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1168
*Boot into the last working kernel and briefly recheck if the feature that
1169
regressed really works.* [:ref:`...<bootworking_bissbs>`]
1171
This will make later steps that cover creating and trimming the configuration do
1172
the right thing.
1174
[:ref:`back to step-by-step guide <bootworking_bissbs>`]
1176
.. _diskspace_bisref:
1178
Space requirements
1179
~~~~~~~~~~~~~~~~~~
1181
*Ensure to have enough free space for building Linux.*
1182
[:ref:`... <diskspace_bissbs>`]
1184
The numbers mentioned are rough estimates with a big extra charge to be on the
1185
safe side, so often you will need less.
1187
If you have space constraints, be sure to hay attention to the :ref:`step about
1188
debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference
1189
section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk
1190
space by quite a few gigabytes.
1192
[:ref:`back to step-by-step guide <diskspace_bissbs>`]
1194
.. _rangecheck_bisref:
1196
Bisection range
1197
~~~~~~~~~~~~~~~
1199
*Determine the kernel versions considered 'good' and 'bad' throughout this
1200
guide.* [:ref:`...<rangecheck_bissbs>`]
1202
Establishing the range of commits to be checked is mostly straightforward,
1203
except when a regression occurred when switching from a release of one stable
1204
series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that case
1205
Git will need some hand holding, as there is no straight line of descent.
1207
That's because with the release of 6.0 mainline carried on to 6.1 while the
1208
stable series 6.0.y branched to the side. It's therefore theoretically possible
1209
that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed by a
1210
commit that went into one of the 6.0.y releases, but never hit mainline or the
1211
6.1.y series. Thankfully that normally should not happen due to the way the
1212
stable/longterm maintainers maintain the code. It's thus pretty safe to assume
1213
6.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel
1214
will be built and tested in the segment '2' of this guide; Git would force you
1215
to do this as well, if you tried bisecting between 6.0.13 and 6.1.15.
1217
[:ref:`back to step-by-step guide <rangecheck_bissbs>`]
1219
.. _buildrequires_bisref:
1221
Install build requirements
1222
~~~~~~~~~~~~~~~~~~~~~~~~~~
1224
*Install all software required to build a Linux kernel.*
1225
[:ref:`...<buildrequires_bissbs>`]
1227
The kernel is pretty stand-alone, but besides tools like the compiler you will
1228
sometimes need a few libraries to build one. How to install everything needed
1229
depends on your Linux distribution and the configuration of the kernel you are
1230
about to build.
1232
Here are a few examples what you typically need on some mainstream
1233
distributions:
1235
* Arch Linux and derivatives::
1237
sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \
1238
pahole perl zlib ncurses qt6-base
1240
* Debian, Ubuntu, and derivatives::
1242
sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \
1243
libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \
1244
libncurses-dev qt6-base-dev g++
1246
* Fedora and derivatives::
1248
sudo dnf install binutils \
1249
/usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \
1250
/usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction}
1252
* openSUSE and derivatives::
1254
sudo zypper install bc binutils bison dwarves flex gcc git \
1255
kernel-install-tools libelf-devel make modutils openssl openssl-devel \
1256
perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel
1258
These commands install a few packages that are often, but not always needed. You
1259
for example might want to skip installing the development headers for ncurses,
1260
which you will only need in case you later might want to adjust the kernel build
1261
configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit
1262
the headers of Qt6 if you do not plan to adjust the .config using 'xconfig'.
1264
You furthermore might need additional libraries and their development headers
1265
for tasks not covered in this guide -- for example when building utilities from
1266
the kernel's tools/ directory.
1268
[:ref:`back to step-by-step guide <buildrequires_bissbs>`]
1270
.. _sources_bisref:
1272
Download the sources using Git
1273
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1275
*Retrieve the Linux mainline sources.*
1276
[:ref:`...<sources_bissbs>`]
1278
The step-by-step guide outlines how to download the Linux sources using a full
1279
Git clone of Linus' mainline repository. There is nothing more to say about
1280
that -- but there are two alternatives ways to retrieve the sources that might
1281
work better for you:
1283
* If you have an unreliable internet connection, consider
1284
:ref:`using a 'Git bundle'<sources_bundle_bisref>`.
1286
* If downloading the complete repository would take too long or requires too
1287
much storage space, consider :ref:`using a 'shallow
1288
clone'<sources_shallow_bisref>`.
1290
.. _sources_bundle_bisref:
1292
Downloading Linux mainline sources using a bundle
1293
"""""""""""""""""""""""""""""""""""""""""""""""""
1295
Use the following commands to retrieve the Linux mainline sources using a
1296
bundle::
1298
wget -c \
1299
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
1300
git clone --no-checkout clone.bundle ~/linux/
1301
cd ~/linux/
1302
git remote remove origin
1303
git remote add mainline \
1304
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
1305
git fetch mainline
1306
git remote add -t master stable \
1307
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1309
In case the 'wget' command fails, just re-execute it, it will pick up where
1310
it left off.
1312
[:ref:`back to step-by-step guide <sources_bissbs>`]
1313
[:ref:`back to section intro <sources_bisref>`]
1315
.. _sources_shallow_bisref:
1317
Downloading Linux mainline sources using a shallow clone
1318
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1320
First, execute the following command to retrieve the latest mainline codebase::
1322
git clone -o mainline --no-checkout --depth 1 -b master \
1323
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
1324
cd ~/linux/
1325
git remote add -t master stable \
1326
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1328
Now deepen your clone's history to the second predecessor of the mainline
1329
release of your 'good' version. In case the latter are 6.0 or 6.0.13, 5.19 would
1330
be the first predecessor and 5.18 the second -- hence deepen the history up to
1331
that version::
1333
git fetch --shallow-exclude=v5.18 mainline
1335
Afterwards add the stable Git repository as remote and all required stable
1336
branches as explained in the step-by-step guide.
1338
Note, shallow clones have a few peculiar characteristics:
1340
* For bisections the history needs to be deepened a few mainline versions
1341
farther than it seems necessary, as explained above already. That's because
1342
Git otherwise will be unable to revert or describe most of the commits within
1343
a range (say 6.1..6.2), as they are internally based on earlier kernels
1344
releases (like 6.0-rc2 or 5.19-rc3).
1346
* This document in most places uses ``git fetch`` with ``--shallow-exclude=``
1347
to specify the earliest version you care about (or to be precise: its git
1348
tag). You alternatively can use the parameter ``--shallow-since=`` to specify
1349
an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to
1350
define the depth of the history you want to download. When using them while
1351
bisecting mainline, ensure to deepen the history to at least 7 months before
1352
the release of the mainline release your 'good' kernel is based on.
1354
* Be warned, when deepening your clone you might encounter an error like
1355
'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'.
1356
In that case run ``git repack -d`` and try again.
1358
[:ref:`back to step-by-step guide <sources_bissbs>`]
1359
[:ref:`back to section intro <sources_bisref>`]
1361
.. _oldconfig_bisref:
1363
Start defining the build configuration for your kernel
1364
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1366
*Start preparing a kernel build configuration (the '.config' file).*
1367
[:ref:`... <oldconfig_bissbs>`]
1369
*Note, this is the first of multiple steps in this guide that create or modify
1370
build artifacts. The commands used in this guide store them right in the source
1371
tree to keep things simple. In case you prefer storing the build artifacts
1372
separately, create a directory like '~/linux-builddir/' and add the parameter
1373
``O=~/linux-builddir/`` to all make calls used throughout this guide. You will
1374
have to point other commands there as well -- among them the ``./scripts/config
1375
[...]`` commands, which will require ``--file ~/linux-builddir/.config`` to
1376
locate the right build configuration.*
1378
Two things can easily go wrong when creating a .config file as advised:
1380
* The oldconfig target will use a .config file from your build directory, if
1381
one is already present there (e.g. '~/linux/.config'). That's totally fine if
1382
that's what you intend (see next step), but in all other cases you want to
1383
delete it. This for example is important in case you followed this guide
1384
further, but due to problems come back here to redo the configuration from
1385
scratch.
1387
* Sometimes olddefconfig is unable to locate the .config file for your running
1388
kernel and will use defaults, as briefly outlined in the guide. In that case
1389
check if your distribution ships the configuration somewhere and manually put
1390
it in the right place (e.g. '~/linux/.config') if it does. On distributions
1391
where /proc/config.gz exists this can be achieved using this command::
1393
zcat /proc/config.gz > .config
1395
Once you put it there, run ``make olddefconfig`` again to adjust it to the
1396
needs of the kernel about to be built.
1398
Note, the olddefconfig target will set any undefined build options to their
1399
default value. If you prefer to set such configuration options manually, use
1400
``make oldconfig`` instead. Then for each undefined configuration option you
1401
will be asked how to proceed; in case you are unsure what to answer, simply hit
1402
'enter' to apply the default value. Note though that for bisections you normally
1403
want to go with the defaults, as you otherwise might enable a new feature that
1404
causes a problem looking like regressions (for example due to security
1405
restrictions).
1407
Occasionally odd things happen when trying to use a config file prepared for one
1408
kernel (say 6.1) on an older mainline release -- especially if it is much older
1409
(say 5.15). That's one of the reasons why the previous step in the guide told
1410
you to boot the kernel where everything works. If you manually add a .config
1411
file you thus want to ensure it's from the working kernel and not from a one
1412
that shows the regression.
1414
In case you want to build kernels for another machine, locate its kernel build
1415
configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy
1416
that file to the build machine and store it as ~/linux/.config; afterwards run
1417
``make olddefconfig`` to adjust it.
1419
[:ref:`back to step-by-step guide <oldconfig_bissbs>`]
1421
.. _localmodconfig_bisref:
1423
Trim the build configuration for your kernel
1424
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1426
*Disable any kernel modules apparently superfluous for your setup.*
1427
[:ref:`... <localmodconfig_bissbs>`]
1429
As explained briefly in the step-by-step guide already: with localmodconfig it
1430
can easily happen that your self-built kernels will lack modules for tasks you
1431
did not perform at least once before utilizing this make target. That happens
1432
when a task requires kernel modules which are only autoloaded when you execute
1433
it for the first time. So when you never performed that task since starting your
1434
kernel the modules will not have been loaded -- and from localmodconfig's point
1435
of view look superfluous, which thus disables them to reduce the amount of code
1436
to be compiled.
1438
You can try to avoid this by performing typical tasks that often will autoload
1439
additional kernel modules: start a VM, establish VPN connections, loop-mount a
1440
CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external
1441
devices (2FA keys, headsets, webcams, ...) as well as storage devices with file
1442
systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it
1443
is hard to think of everything that might be needed -- even kernel developers
1444
often forget one thing or another at this point.
1446
Do not let that risk bother you, especially when compiling a kernel only for
1447
testing purposes: everything typically crucial will be there. And if you forget
1448
something important you can turn on a missing feature manually later and quickly
1449
run the commands again to compile and install a kernel that has everything you
1450
need.
1452
But if you plan to build and use self-built kernels regularly, you might want to
1453
reduce the risk by recording which modules your system loads over the course of
1454
a few weeks. You can automate this with `modprobed-db
1455
<https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to
1456
point localmodconfig to the list of modules modprobed-db noticed being used::
1458
yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig
1460
That parameter also allows you to build trimmed kernels for another machine in
1461
case you copied a suitable .config over to use as base (see previous step). Just
1462
run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to
1463
your build's host home directory. Then run these commands instead of the one the
1464
step-by-step guide mentions::
1466
yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig
1468
[:ref:`back to step-by-step guide <localmodconfig_bissbs>`]
1470
.. _tagging_bisref:
1472
Tag the kernels about to be build
1473
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1475
*Ensure all the kernels you will build are clearly identifiable using a
1476
special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`]
1478
This allows you to differentiate your distribution's kernels from those created
1479
during this process, as the file or directories for the latter will contain
1480
'-local' in the name; it also helps picking the right entry in the boot menu and
1481
not lose track of you kernels, as their version numbers will look slightly
1482
confusing during the bisection.
1484
[:ref:`back to step-by-step guide <tagging_bissbs>`]
1486
.. _debugsymbols_bisref:
1488
Decide to enable or disable debug symbols
1489
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1491
*Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`]
1493
Having debug symbols available can be important when your kernel throws a
1494
'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be
1495
able to find the exact place where the problem occurred in the code. But
1496
collecting and embedding the needed debug information takes time and consumes
1497
quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel
1498
trimmed with localmodconfig consumed around 5 Gigabyte of space with debug
1499
symbols, but less than 1 when they were disabled. The resulting kernel image and
1500
modules are bigger as well, which increases storage requirements for /boot/ and
1501
load times.
1503
In case you want a small kernel and are unlikely to decode a stack trace later,
1504
you thus might want to disable debug symbols to avoid those downsides. If it
1505
later turns out that you need them, just enable them as shown and rebuild the
1506
kernel.
1508
You on the other hand definitely want to enable them for this process, if there
1509
is a decent chance that you need to decode a stack trace later. The section
1510
'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst
1511
explains this process in more detail.
1513
[:ref:`back to step-by-step guide <debugsymbols_bissbs>`]
1515
.. _configmods_bisref:
1517
Adjust build configuration
1518
~~~~~~~~~~~~~~~~~~~~~~~~~~
1520
*Check if you may want or need to adjust some other kernel configuration
1521
options:*
1523
Depending on your needs you at this point might want or have to adjust some
1524
kernel configuration options.
1526
.. _configmods_distros_bisref:
1528
Distro specific adjustments
1529
"""""""""""""""""""""""""""
1531
*Are you running* [:ref:`... <configmods_bissbs>`]
1533
The following sections help you to avoid build problems that are known to occur
1534
when following this guide on a few commodity distributions.
1536
**Debian:**
1538
* Remove a stale reference to a certificate file that would cause your build to
1539
fail::
1541
./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''
1543
Alternatively, download the needed certificate and make that configuration
1544
option point to it, as `the Debian handbook explains in more detail
1545
<https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_
1546
-- or generate your own, as explained in
1547
Documentation/admin-guide/module-signing.rst.
1549
[:ref:`back to step-by-step guide <configmods_bissbs>`]
1551
.. _configmods_individual_bisref:
1553
Individual adjustments
1554
""""""""""""""""""""""
1556
*If you want to influence the other aspects of the configuration, do so
1557
now.* [:ref:`... <configmods_bissbs>`]
1559
At this point you can use a command like ``make menuconfig`` or ``make nconfig``
1560
to enable or disable certain features using a text-based user interface; to use
1561
a graphical configuration utility, run ``make xconfig`` instead. Both of them
1562
require development libraries from toolkits they are rely on (ncurses
1563
respectively Qt5 or Qt6); an error message will tell you if something required
1564
is missing.
1566
[:ref:`back to step-by-step guide <configmods_bissbs>`]
1568
.. _saveconfig_bisref:
1570
Put the .config file aside
1571
~~~~~~~~~~~~~~~~~~~~~~~~~~
1573
*Reprocess the .config after the latest changes and store it in a safe place.*
1574
[:ref:`... <saveconfig_bissbs>`]
1576
Put the .config you prepared aside, as you want to copy it back to the build
1577
directory every time during this guide before you start building another
1578
kernel. That's because going back and forth between different versions can alter
1579
.config files in odd ways; those occasionally cause side effects that could
1580
confuse testing or in some cases render the result of your bisection
1581
meaningless.
1583
[:ref:`back to step-by-step guide <saveconfig_bissbs>`]
1585
.. _introlatestcheck_bisref:
1587
Try to reproduce the problem with the latest codebase
1588
-----------------------------------------------------
1590
*Verify the regression is not caused by some .config change and check if it
1591
still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`]
1593
For some readers it might seem unnecessary to check the latest codebase at this
1594
point, especially if you did that already with a kernel prepared by your
1595
distributor or face a regression within a stable/longterm series. But it's
1596
highly recommended for these reasons:
1598
* You will run into any problems caused by your setup before you actually begin
1599
a bisection. That will make it a lot easier to differentiate between 'this
1600
most likely is some problem in my setup' and 'this change needs to be skipped
1601
during the bisection, as the kernel sources at that stage contain an unrelated
1602
problem that causes building or booting to fail'.
1604
* These steps will rule out if your problem is caused by some change in the
1605
build configuration between the 'working' and the 'broken' kernel. This for
1606
example can happen when your distributor enabled an additional security
1607
feature in the newer kernel which was disabled or not yet supported by the
1608
older kernel. That security feature might get into the way of something you
1609
do -- in which case your problem from the perspective of the Linux kernel
1610
upstream developers is not a regression, as
1611
Documentation/admin-guide/reporting-regressions.rst explains in more detail.
1612
You thus would waste your time if you'd try to bisect this.
1614
* If the cause for your regression was already fixed in the latest mainline
1615
codebase, you'd perform the bisection for nothing. This holds true for a
1616
regression you encountered with a stable/longterm release as well, as they are
1617
often caused by problems in mainline changes that were backported -- in which
1618
case the problem will have to be fixed in mainline first. Maybe it already was
1619
fixed there and the fix is already in the process of being backported.
1621
* For regressions within a stable/longterm series it's furthermore crucial to
1622
know if the issue is specific to that series or also happens in the mainline
1623
kernel, as the report needs to be sent to different people:
1625
* Regressions specific to a stable/longterm series are the stable team's
1626
responsibility; mainline Linux developers might or might not care.
1628
* Regressions also happening in mainline are something the regular Linux
1629
developers and maintainers have to handle; the stable team does not care
1630
and does not need to be involved in the report, they just should be told
1631
to backport the fix once it's ready.
1633
Your report might be ignored if you send it to the wrong party -- and even
1634
when you get a reply there is a decent chance that developers tell you to
1635
evaluate which of the two cases it is before they take a closer look.
1637
[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]
1639
.. _checkoutmaster_bisref:
1641
Check out the latest Linux codebase
1642
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1644
*Check out the latest Linux codebase.*
1645
[:ref:`... <checkoutmaster_bissbs>`]
1647
In case you later want to recheck if an ever newer codebase might fix the
1648
problem, remember to run that ``git fetch --shallow-exclude [...]`` command
1649
again mentioned earlier to update your local Git repository.
1651
[:ref:`back to step-by-step guide <checkoutmaster_bissbs>`]
1653
.. _build_bisref:
1655
Build your kernel
1656
~~~~~~~~~~~~~~~~~
1658
*Build the image and the modules of your first kernel using the config file
1659
you prepared.* [:ref:`... <build_bissbs>`]
1661
A lot can go wrong at this stage, but the instructions below will help you help
1662
yourself. Another subsection explains how to directly package your kernel up as
1663
deb, rpm or tar file.
1665
Dealing with build errors
1666
"""""""""""""""""""""""""
1668
When a build error occurs, it might be caused by some aspect of your machine's
1669
setup that often can be fixed quickly; other times though the problem lies in
1670
the code and can only be fixed by a developer. A close examination of the
1671
failure messages coupled with some research on the internet will often tell you
1672
which of the two it is. To perform such investigation, restart the build
1673
process like this::
1675
make V=1
1677
The ``V=1`` activates verbose output, which might be needed to see the actual
1678
error. To make it easier to spot, this command also omits the ``-j $(nproc
1679
--all)`` used earlier to utilize every CPU core in the system for the job -- but
1680
this parallelism also results in some clutter when failures occur.
1682
After a few seconds the build process should run into the error again. Now try
1683
to find the most crucial line describing the problem. Then search the internet
1684
for the most important and non-generic section of that line (say 4 to 8 words);
1685
avoid or remove anything that looks remotely system-specific, like your username
1686
or local path names like ``/home/username/linux/``. First try your regular
1687
internet search engine with that string, afterwards search Linux kernel mailing
1688
lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_.
1690
This most of the time will find something that will explain what is wrong; quite
1691
often one of the hits will provide a solution for your problem, too. If you
1692
do not find anything that matches your problem, try again from a different angle
1693
by modifying your search terms or using another line from the error messages.
1695
In the end, most issues you run into have likely been encountered and
1696
reported by others already. That includes issues where the cause is not your
1697
system, but lies in the code. If you run into one of those, you might thus find
1698
a solution (e.g. a patch) or workaround for your issue, too.
1700
Package your kernel up
1701
""""""""""""""""""""""
1703
The step-by-step guide uses the default make targets (e.g. 'bzImage' and
1704
'modules' on x86) to build the image and the modules of your kernel, which later
1705
steps of the guide then install. You instead can also directly build everything
1706
and directly package it up by using one of the following targets:
1708
* ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package
1710
* ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package
1712
* ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball
1714
This is just a selection of available make targets for this purpose, see
1715
``make help`` for others. You can also use these targets after running
1716
``make -j $(nproc --all)``, as they will pick up everything already built.
1718
If you employ the targets to generate deb or rpm packages, ignore the
1719
step-by-step guide's instructions on installing and removing your kernel;
1720
instead install and remove the packages using the package utility for the format
1721
(e.g. dpkg and rpm) or a package management utility build on top of them (apt,
1722
aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using
1723
these two make targets are designed to work on various distributions utilizing
1724
those formats, they thus will sometimes behave differently than your
1725
distribution's kernel packages.
1727
[:ref:`back to step-by-step guide <build_bissbs>`]
1729
.. _install_bisref:
1731
Put the kernel in place
1732
~~~~~~~~~~~~~~~~~~~~~~~
1734
*Install the kernel you just built.* [:ref:`... <install_bissbs>`]
1736
What you need to do after executing the command in the step-by-step guide
1737
depends on the existence and the implementation of ``/sbin/installkernel``
1738
executable on your distribution.
1740
If installkernel is found, the kernel's build system will delegate the actual
1741
installation of your kernel image to this executable, which then performs some
1742
or all of these tasks:
1744
* On almost all Linux distributions installkernel will store your kernel's
1745
image in /boot/, usually as '/boot/vmlinuz-<kernelrelease_id>'; often it will
1746
put a 'System.map-<kernelrelease_id>' alongside it.
1748
* On most distributions installkernel will then generate an 'initramfs'
1749
(sometimes also called 'initrd'), which usually are stored as
1750
'/boot/initramfs-<kernelrelease_id>.img' or
1751
'/boot/initrd-<kernelrelease_id>'. Commodity distributions rely on this file
1752
for booting, hence ensure to execute the make target 'modules_install' first,
1753
as your distribution's initramfs generator otherwise will be unable to find
1754
the modules that go into the image.
1756
* On some distributions installkernel will then add an entry for your kernel
1757
to your bootloader's configuration.
1759
You have to take care of some or all of the tasks yourself, if your
1760
distribution lacks an installkernel script or does only handle part of them.
1761
Consult the distribution's documentation for details. If in doubt, install the
1762
kernel manually::
1764
sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
1765
sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)
1767
Now generate your initramfs using the tools your distribution provides for this
1768
process. Afterwards add your kernel to your bootloader configuration and reboot.
1770
[:ref:`back to step-by-step guide <install_bissbs>`]
1772
.. _storagespace_bisref:
1774
Storage requirements per kernel
1775
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1777
*Check how much storage space the kernel, its modules, and other related files
1778
like the initramfs consume.* [:ref:`... <storagespace_bissbs>`]
1780
The kernels built during a bisection consume quite a bit of space in /boot/ and
1781
/lib/modules/, especially if you enabled debug symbols. That makes it easy to
1782
fill up volumes during a bisection -- and due to that even kernels which used to
1783
work earlier might fail to boot. To prevent that you will need to know how much
1784
space each installed kernel typically requires.
1786
Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in
1787
the guide will match all files needed to boot your kernel -- but neither the
1788
path nor the naming scheme are mandatory. On some distributions you thus will
1789
need to look in different places.
1791
[:ref:`back to step-by-step guide <storagespace_bissbs>`]
1793
.. _tainted_bisref:
1795
Check if your newly built kernel considers itself 'tainted'
1796
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1798
*Check if the kernel marked itself as 'tainted'.*
1799
[:ref:`... <tainted_bissbs>`]
1801
Linux marks itself as tainted when something happens that potentially leads to
1802
follow-up errors that look totally unrelated. That is why developers might
1803
ignore or react scantly to reports from tainted kernels -- unless of course the
1804
kernel set the flag right when the reported bug occurred.
1806
That's why you want check why a kernel is tainted as explained in
1807
Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own
1808
interest, as your testing might be flawed otherwise.
1810
[:ref:`back to step-by-step guide <tainted_bissbs>`]
1812
.. _recheckbroken_bisref:
1814
Check the kernel built from a recent mainline codebase
1815
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1817
*Verify if your bug occurs with the newly built kernel.*
1818
[:ref:`... <recheckbroken_bissbs>`]
1820
There are a couple of reasons why your bug or regression might not show up with
1821
the kernel you built from the latest codebase. These are the most frequent:
1823
* The bug was fixed meanwhile.
1825
* What you suspected to be a regression was caused by a change in the build
1826
configuration the provider of your kernel carried out.
1828
* Your problem might be a race condition that does not show up with your kernel;
1829
the trimmed build configuration, a different setting for debug symbols, the
1830
compiler used, and various other things can cause this.
1832
* In case you encountered the regression with a stable/longterm kernel it might
1833
be a problem that is specific to that series; the next step in this guide will
1834
check this.
1836
[:ref:`back to step-by-step guide <recheckbroken_bissbs>`]
1838
.. _recheckstablebroken_bisref:
1840
Check the kernel built from the latest stable/longterm codebase
1841
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1843
*Are you facing a regression within a stable/longterm release, but failed to
1844
reproduce it with the kernel you just built using the latest mainline sources?
1845
Then check if the latest codebase for the particular series might already fix
1846
the problem.* [:ref:`... <recheckstablebroken_bissbs>`]
1848
If this kernel does not show the regression either, there most likely is no need
1849
for a bisection.
1851
[:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`]
1853
.. _introworkingcheck_bisref:
1855
Ensure the 'good' version is really working well
1856
------------------------------------------------
1858
*Check if the kernels you build work fine.*
1859
[:ref:`... <introworkingcheck_bissbs>`]
1861
This section will reestablish a known working base. Skipping it might be
1862
appealing, but is usually a bad idea, as it does something important:
1864
It will ensure the .config file you prepared earlier actually works as expected.
1865
That is in your own interest, as trimming the configuration is not foolproof --
1866
and you might be building and testing ten or more kernels for nothing before
1867
starting to suspect something might be wrong with the build configuration.
1869
That alone is reason enough to spend the time on this, but not the only reason.
1871
Many readers of this guide normally run kernels that are patched, use add-on
1872
modules, or both. Those kernels thus are not considered 'vanilla' -- therefore
1873
it's possible that the thing that regressed might never have worked in vanilla
1874
builds of the 'good' version in the first place.
1876
There is a third reason for those that noticed a regression between
1877
stable/longterm kernels of different series (e.g. 6.0.13..6.1.5): it will
1878
ensure the kernel version you assumed to be 'good' earlier in the process (e.g.
1879
6.0) actually is working.
1881
[:ref:`back to step-by-step guide <introworkingcheck_bissbs>`]
1883
.. _recheckworking_bisref:
1885
Build your own version of the 'good' kernel
1886
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1888
*Build your own variant of the working kernel and check if the feature that
1889
regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`]
1891
In case the feature that broke with newer kernels does not work with your first
1892
self-built kernel, find and resolve the cause before moving on. There are a
1893
multitude of reasons why this might happen. Some ideas where to look:
1895
* Check the taint status and the output of ``dmesg``, maybe something unrelated
1896
went wrong.
1898
* Maybe localmodconfig did something odd and disabled the module required to
1899
test the feature? Then you might want to recreate a .config file based on the
1900
one from the last working kernel and skip trimming it down; manually disabling
1901
some features in the .config might work as well to reduce the build time.
1903
* Maybe it's not a kernel regression and something that is caused by some fluke,
1904
a broken initramfs (also known as initrd), new firmware files, or an updated
1905
userland software?
1907
* Maybe it was a feature added to your distributor's kernel which vanilla Linux
1908
at that point never supported?
1910
Note, if you found and fixed problems with the .config file, you want to use it
1911
to build another kernel from the latest codebase, as your earlier tests with
1912
mainline and the latest version from an affected stable/longterm series were
1913
most likely flawed.
1915
[:ref:`back to step-by-step guide <recheckworking_bissbs>`]
1917
Perform a bisection and validate the result
1918
-------------------------------------------
1920
*With all the preparations and precaution builds taken care of, you are now
1921
ready to begin the bisection.* [:ref:`... <introbisect_bissbs>`]
1923
The steps in this segment perform and validate the bisection.
1925
[:ref:`back to step-by-step guide <introbisect_bissbs>`].
1927
.. _bisectstart_bisref:
1929
Start the bisection
1930
~~~~~~~~~~~~~~~~~~~
1932
*Start the bisection and tell Git about the versions earlier established as
1933
'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`]
1935
This will start the bisection process; the last of the commands will make Git
1936
check out a commit round about half-way between the 'good' and the 'bad' changes
1937
for you to test.
1939
[:ref:`back to step-by-step guide <bisectstart_bissbs>`]
1941
.. _bisectbuild_bisref:
1943
Build a kernel from the bisection point
1944
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1946
*Build, install, and boot a kernel from the code Git checked out using the
1947
same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`]
1949
There are two things worth of note here:
1951
* Occasionally building the kernel will fail or it might not boot due some
1952
problem in the code at the bisection point. In that case run this command::
1954
git bisect skip
1956
Git will then check out another commit nearby which with a bit of luck should
1957
work better. Afterwards restart executing this step.
1959
* Those slightly odd looking version identifiers can happen during bisections,
1960
because the Linux kernel subsystems prepare their changes for a new mainline
1961
release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus
1962
base them on a somewhat earlier point like 6.1-rc1 or even 6.0 -- and then
1963
get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This
1964
leads to those slightly odd looking version identifiers coming up during
1965
bisections.
1967
[:ref:`back to step-by-step guide <bisectbuild_bissbs>`]
1969
.. _bisecttest_bisref:
1971
Bisection checkpoint
1972
~~~~~~~~~~~~~~~~~~~~
1974
*Check if the feature that regressed works in the kernel you just built.*
1975
[:ref:`... <bisecttest_bissbs>`]
1977
Ensure what you tell Git is accurate: getting it wrong just one time will bring
1978
the rest of the bisection totally off course, hence all testing after that point
1979
will be for nothing.
1981
[:ref:`back to step-by-step guide <bisecttest_bissbs>`]
1983
.. _bisectlog_bisref:
1985
Put the bisection log away
1986
~~~~~~~~~~~~~~~~~~~~~~~~~~
1988
*Store Git's bisection log and the current .config file in a safe place.*
1989
[:ref:`... <bisectlog_bissbs>`]
1991
As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will
1992
render the end result of a bisection useless. In that case you'd normally have
1993
to restart the bisection from scratch. The log can prevent that, as it might
1994
allow someone to point out where a bisection likely went sideways -- and then
1995
instead of testing ten or more kernels you might only have to build a few to
1996
resolve things.
1998
The .config file is put aside, as there is a decent chance that developers might
1999
ask for it after you report the regression.
2001
[:ref:`back to step-by-step guide <bisectlog_bissbs>`]
2003
.. _revert_bisref:
2005
Try reverting the culprit
2006
~~~~~~~~~~~~~~~~~~~~~~~~~
2008
*Try reverting the culprit on top of the latest codebase to see if this fixes
2009
your regression.* [:ref:`... <revert_bissbs>`]
2011
This is an optional step, but whenever possible one you should try: there is a
2012
decent chance that developers will ask you to perform this step when you bring
2013
the bisection result up. So give it a try, you are in the flow already, building
2014
one more kernel shouldn't be a big deal at this point.
2016
The step-by-step guide covers everything relevant already except one slightly
2017
rare thing: did you bisected a regression that also happened with mainline using
2018
a stable/longterm series, but Git failed to revert the commit in mainline? Then
2019
try to revert the culprit in the affected stable/longterm series -- and if that
2020
succeeds, test that kernel version instead.
2022
[:ref:`back to step-by-step guide <revert_bissbs>`]
2024
Cleanup steps during and after following this guide
2025
---------------------------------------------------
2027
*During and after following this guide you might want or need to remove some
2028
of the kernels you installed.* [:ref:`... <introclosure_bissbs>`]
2030
The steps in this section describe clean-up procedures.
2032
[:ref:`back to step-by-step guide <introclosure_bissbs>`].
2034
.. _makeroom_bisref:
2036
Cleaning up during the bisection
2037
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2039
*To remove one of the kernels you installed, look up its 'kernelrelease'
2040
identifier.* [:ref:`... <makeroom_bissbs>`]
2042
The kernels you install during this process are easy to remove later, as its
2043
parts are only stored in two places and clearly identifiable. You thus do not
2044
need to worry to mess up your machine when you install a kernel manually (and
2045
thus bypass your distribution's packaging system): all parts of your kernels are
2046
relatively easy to remove later.
2048
One of the two places is a directory in /lib/modules/, which holds the modules
2049
for each installed kernel. This directory is named after the kernel's release
2050
identifier; hence, to remove all modules for one of the kernels you built,
2051
simply remove its modules directory in /lib/modules/.
2053
The other place is /boot/, where typically two up to five files will be placed
2054
during installation of a kernel. All of them usually contain the release name in
2055
their file name, but how many files and their exact names depend somewhat on
2056
your distribution's installkernel executable and its initramfs generator. On
2057
some distributions the ``kernel-install remove...`` command mentioned in the
2058
step-by-step guide will delete all of these files for you while also removing
2059
the menu entry for the kernel from your bootloader configuration. On others you
2060
have to take care of these two tasks yourself. The following command should
2061
interactively remove the three main files of a kernel with the release name
2062
'6.0-rc1-local-gcafec0cacaca0'::
2064
rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0
2066
Afterwards check for other files in /boot/ that have
2067
'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.
2068
Now remove the boot entry for the kernel from your bootloader's configuration;
2069
the steps to do that vary quite a bit between Linux distributions.
2071
Note, be careful with wildcards like '*' when deleting files or directories
2072
for kernels manually: you might accidentally remove files of a 6.0.13 kernel
2073
when all you want is to remove 6.0 or 6.0.1.
2075
[:ref:`back to step-by-step guide <makeroom_bissbs>`]
2077
Cleaning up after the bisection
2078
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2080
.. _finishingtouch_bisref:
2082
*Once you have finished the bisection, do not immediately remove anything
2083
you set up, as you might need a few things again.*
2084
[:ref:`... <finishingtouch_bissbs>`]
2086
When you are really short of storage space removing the kernels as described in
2087
the step-by-step guide might not free as much space as you would like. In that
2088
case consider running ``rm -rf ~/linux/*`` as well now. This will remove the
2089
build artifacts and the Linux sources, but will leave the Git repository
2090
(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the
2091
sources back.
2093
Removing the repository as well would likely be unwise at this point: there
2094
is a decent chance developers will ask you to build another kernel to
2095
perform additional tests -- like testing a debug patch or a proposed fix.
2096
Details on how to perform those can be found in the section :ref:`Optional
2097
tasks: test reverts, patches, or later versions <introoptional_bissbs>`.
2099
Additional tests are also the reason why you want to keep the
2100
~/kernel-config-working file around for a few weeks.
2102
[:ref:`back to step-by-step guide <finishingtouch_bissbs>`]
2104
.. _introoptional_bisref:
2106
Test reverts, patches, or later versions
2107
----------------------------------------
2109
*While or after reporting a bug, you might want or potentially will be asked
2110
to test reverts, patches, proposed fixes, or other versions.*
2111
[:ref:`... <introoptional_bissbs>`]
2113
All the commands used in this section should be pretty straight forward, so
2114
there is not much to add except one thing: when setting a kernel tag as
2115
instructed, ensure it is not much longer than the one used in the example, as
2116
problems will arise if the kernelrelease identifier exceeds 63 characters.
2118
[:ref:`back to step-by-step guide <introoptional_bissbs>`].
2121
Additional information
2122
======================
2124
.. _buildhost_bis:
2126
Build kernels on a different machine
2127
------------------------------------
2129
To compile kernels on another system, slightly alter the step-by-step guide's
2130
instructions:
2132
* Start following the guide on the machine where you want to install and test
2133
the kernels later.
2135
* After executing ':ref:`Boot into the working kernel and briefly use the
2136
apparently broken feature <bootworking_bissbs>`', save the list of loaded
2137
modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the
2138
build configuration for the running kernel (see ':ref:`Start defining the
2139
build configuration for your kernel <oldconfig_bisref>`' for hints on where
2140
to find it) and store it as '~/test-machine-config-working'. Transfer both
2141
files to the home directory of your build host.
2143
* Continue the guide on the build host (e.g. with ':ref:`Ensure to have enough
2144
free space for building [...] <diskspace_bissbs>`').
2146
* When you reach ':ref:`Start preparing a kernel build configuration[...]
2147
<oldconfig_bissbs>`': before running ``make olddefconfig`` for the first time,
2148
execute the following command to base your configuration on the one from the
2149
test machine's 'working' kernel::
2151
cp ~/test-machine-config-working ~/linux/.config
2153
* During the next step to ':ref:`disable any apparently superfluous kernel
2154
modules <localmodconfig_bissbs>`' use the following command instead::
2156
yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig
2158
* Continue the guide, but ignore the instructions outlining how to compile,
2159
install, and reboot into a kernel every time they come up. Instead build
2160
like this::
2162
cp ~/kernel-config-working .config
2163
make olddefconfig &&
2164
make -j $(nproc --all) targz-pkg
2166
This will generate a gzipped tar file whose name is printed in the last
2167
line shown; for example, a kernel with the kernelrelease identifier
2168
'6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will
2169
be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'.
2171
Copy that file to your test machine's home directory.
2173
* Switch to the test machine to check if you have enough space to hold another
2174
kernel. Then extract the file you transferred::
2176
sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C /
2178
Afterwards :ref:`generate the initramfs and add the kernel to your boot
2179
loader's configuration <install_bisref>`; on some distributions the following
2180
command will take care of both these tasks::
2182
sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa423
2184
Now reboot and ensure you started the intended kernel.
2186
This approach even works when building for another architecture: just install
2187
cross-compilers and add the appropriate parameters to every invocation of make
2188
(e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]``).
2190
Additional reading material
2191
---------------------------
2193
* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
2194
`fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
2195
in the Git documentation.
2196
* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
2197
from kernel developer Nathan Chancellor.
2198
* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
2199
* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
2201
..
2202
end-of-content
2203
..
2204
This document is maintained by Thorsten Leemhuis <[email protected]>. If
2205
you spot a typo or small mistake, feel free to let him know directly and
2206
he'll fix it. You are free to do the same in a mostly informal way if you
2207
want to contribute changes to the text -- but for copyright reasons please CC
2208
[email protected] and 'sign-off' your contribution as
2209
Documentation/process/submitting-patches.rst explains in the section 'Sign
2210
your work - the Developer's Certificate of Origin'.
2211
..
2212
This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
2213
of the file. If you want to distribute this text under CC-BY-4.0 only,
2214
please use 'The Linux kernel development community' for author attribution
2215
and link this as source:
2216
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
2218
..
2219
Note: Only the content of this RST file as found in the Linux kernel sources
2220
is available under CC-BY-4.0, as versions of this text that were processed
2221
(for example by the kernel's build system) might contain content taken from
2222
files which use a more restrictive license.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
How to verify bugs and bisect regressions
1-27이 문서는 Linux 커널 문제가 개발자가 현재 지원하는 코드에서도 발생하는지 확인하고, 이전 버전에서는 없던 회귀라면 문제를 일으킨 변경을 찾아내는 방법을 설명합니다. 라이선스는 원문 상단의 SPDX 식별자 `(GPL-2.0+ OR CC-BY-4.0)`를 따릅니다.
대상 독자는 일반 하드웨어에서 주류 Linux 배포판 커널을 사용하다가 upstream Linux 개발자에게 커널 버그를 보고하려는 사람입니다. 이미 커널 빌드에 익숙한 사용자에게도 같은 절차가 유용하며, 숙련된 개발자도 가끔 범하는 실수를 피하도록 돕습니다.
RST 원본보다 탐색하기 쉬운 렌더링 문서는 `https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html`에서 볼 수 있습니다. 특히 참조 절을 확인한 뒤 이전 위치로 돌아올 때 렌더링 버전이 편리합니다.
The essence of the process (aka 'TL;DR')
28-221Linux를 직접 빌드하거나 bisect해 본 적이 없다면 이 요약을 건너뛰고 아래의 단계별 안내를 따르는 편이 좋습니다. 단계별 안내는 같은 명령을 짧게 설명하면서 대안, 함정, 현재 사례에 필요할 수 있는 추가 사항을 참조 절과 함께 제시합니다.
현재 지원되는 코드에 버그가 있는지만 확인하려면 준비 단계와 Segment 1만 수행합니다. 이때 평소 사용하는 최신 커널을 'working' 커널로 간주합니다. 예시는 6.0을 working 버전으로 보고 그 소스로 `.config`를 준비합니다.
회귀를 조사한다면 적어도 Segment 2 끝까지 진행한 뒤 예비 보고를 제출할 수 있습니다. 완전한 회귀 보고에 필요한 bisect까지 하려면 Segment 3도 수행합니다. 예시는 6.0.13을 마지막 working 커널, 6.1.5를 처음 broken 커널로 보고, 6.0 릴리스를 'good'으로 삼아 `.config`를 만듭니다.
준비 단계에서는 외부 유지보수 커널 모듈에 의존하거나 부팅 중 모듈을 자동 빌드하는 소프트웨어를 제거하고, Secure Boot가 직접 빌드한 커널을 허용하게 하며, working 커널로 부팅합니다. 빌드 도구와 약 15GB의 여유 공간을 준비한 뒤 다음 명령으로 소스와 구성을 마련합니다.
# * Remove any software that depends on externally maintained kernel modules
# or builds any automatically during bootup.
# * Ensure Secure Boot permits booting self-compiled Linux kernels.
# * If you are not already running the 'working' kernel, reboot into it.
# * Install compilers and everything else needed for building Linux.
# * Ensure to have 15 Gigabyte free space in your home directory.
git clone -o mainline --no-checkout \
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
cd ~/linux/
git remote add -t master stable \
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
git switch --detach v6.0
# * Hint: if you used an existing clone, ensure no stale .config is around.
make olddefconfig
# * Ensure the former command picked the .config of the 'working' kernel.
# * Connect external hardware (USB keys, tokens, ...), start a VM, bring up
# VPNs, mount network shares, and briefly try the feature that is broken.
yes '' | make localmodconfig
./scripts/config --set-str CONFIG_LOCALVERSION '-local'
./scripts/config -e CONFIG_LOCALVERSION_AUTO
# * Note, when short on storage space, check the guide for an alternative:
./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
-e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
# * Hint: at this point you might want to adjust the build configuration;
# you'll have to, if you are running Debian.
make olddefconfig
cp .config ~/kernel-config-workingSegment 1은 최신 mainline 코드로 커널을 빌드합니다. 이미 수정되었는지와 어느 개발자에게 알려야 하는지를 확인하고, 회귀라면 `.config` 변경이 원인일 가능성도 배제합니다. 먼저 최신 mainline을 checkout합니다.
cd ~/linux/
git switch --discard-changes --detach mainline/master준비한 구성으로 빌드하고 설치한 뒤 부팅합니다. `/boot/`와 `/lib/modules/`의 공간을 확인하고, 배포판에 따라 `installkernel`이 일부 작업을 하지 않을 수 있다는 점을 유의합니다. 설치 크기와 kernelrelease를 기록하고, 재부팅 뒤 `uname -r`과 taint 값을 확인합니다.
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Make sure there is enough disk space to hold another kernel:
df -h /boot/ /lib/modules/
# * Note: on Arch Linux, its derivatives and a few other distributions
# the following commands will do nothing at all or only part of the
# job. See the step-by-step guide for further details.
sudo make modules_install
command -v installkernel && sudo make install
# * Check how much space your self-built kernel actually needs, which
# enables you to make better estimates later:
du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
du -sh /lib/modules/$(make -s kernelrelease)/
# * Hint: the output of the following command will help you pick the
# right kernel from the boot menu:
make -s kernelrelease | tee -a ~/kernels-built
reboot
# * Once booted, ensure you are running the kernel you just built by
# checking if the output of the next two commands matches:
tail -n 1 ~/kernels-built
uname -r
cat /proc/sys/kernel/tainted새 커널에서도 문제가 발생하는지 검사합니다. Segment 2에서는 잘라낸 `.config`가 실제로 작동하는지 확인하기 위해 good 버전 소스를 checkout합니다.
cd ~/linux/
git switch --discard-changes --detach v6.0Segment 1의 빌드·설치·부팅 절차를 반복하되 이미 대략적인 크기를 알았으므로 `du` 명령은 생략해도 됩니다. broken 커널에서 회귀한 기능이 이 커널에서는 실제로 정상인지 확인합니다.
Segment 3에서는 bad 버전이 속한 stable branch를 받아 bisect를 시작합니다.
git remote set-branches --add stable linux-6.1.y
git fetch stablecd ~/linux/
git bisect start
git bisect good v6.0
git bisect bad v6.1.5각 bisect 지점에서 커널을 빌드·설치·부팅합니다. 무관한 빌드나 부팅 실패는 `git bisect skip`, 기능이 정상이면 `git bisect good`, 재현되면 `git bisect bad`를 실행합니다. Git이 다음 commit을 checkout하므로 최초 bad commit을 찾을 때까지 반복하며, 저장 공간이 부족하면 아래 cleanup 절을 사용합니다.
bisect가 끝나면 로그와 culprit 시점의 구성을 보관하고 상태를 초기화합니다.
cd ~/linux/
git bisect log > ~/bisect-log
cp .config ~/bisection-config-culprit
git bisect reset가능하면 최신 mainline 위에서 culprit를 revert해 결과를 검증합니다. 일부 commit은 revert할 수 없지만, 명령이 성공하면 기본 `.config`를 다시 복사하지 말고 커널을 한 번 더 빌드·설치·부팅해 회귀가 사라지는지 확인합니다.
git switch --discard-changes --detach mainline/master
git revert --no-edit cafec0cacaca0
cp ~/kernel-config-working .config
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'bisect 중 공간이 부족하면 Segment 1과 2에서 만든 커널은 당분간 남기고, Segment 3에서 시험한 오래된 커널부터 kernelrelease 순서로 찾습니다.
ls -ltr /lib/modules/*-local*예를 들어 `6.0-rc1-local-gcafec0cacaca0` 커널은 다음처럼 제거합니다.
sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
# * Note, on some distributions kernel-install is missing
# or does only part of the job.bisect 결과를 성공적으로 검증했다면 실제 bisect 중 만든 커널은 제거해도 되지만, 그 전후에 만든 기준 커널은 1~2주 보관하는 편이 좋습니다.
나중에 debug patch나 제안된 수정안을 시험할 때는 최신 mainline을 받고 patch를 적용한 뒤, 저장한 구성을 복사하고 식별 가능한 `CONFIG_LOCALVERSION`을 지정합니다.
git fetch mainline
git switch --discard-changes --detach mainline/master
git apply /tmp/foobars-proposed-fix-v1.patch
cp ~/kernel-config-working .config
./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'그 뒤 Segment 1의 빌드·설치·부팅 절차를 반복하되, 이미 구성을 복사했으므로 첫 번째 `.config` 복사 명령은 생략합니다.
Step-by-step guide on how to verify bugs and bisect regressions
222-273이 안내서는 보고하려는 버그나 회귀를 조사하기 위해 직접 Linux 커널을 준비하는 방법을 설명합니다. 현재 지원되는 코드에서도 문제가 있는지 확인하려면 Segment 1 끝까지 수행합니다. 이전 버전에서는 문제가 없었다면 우선 처리 대상인 회귀인지 확인하기 위해 Segment 2까지 진행해야 합니다.
결과에 따라 일반 버그 보고나 예비 회귀 보고를 제출할 수 있으며, 곧바로 Segment 3에서 bisect를 수행해 개발자가 대응해야 하는 완전한 회귀 보고를 만들 수도 있습니다. 전체 흐름은 준비, 최신 코드 검사, 빌드된 커널 검증, bisect와 결과 검증, cleanup, 선택적 revert·patch·후속 버전 시험 순서입니다.
각 Segment는 핵심 절차를 보여주고, 종합 참조 절은 거의 모든 단계의 대안과 함정, 발생 가능한 문제와 복구 방법을 제공합니다. 보고 방법은 `Documentation/admin-guide/reporting-issues.rst`, 회귀의 정의는 `Documentation/admin-guide/reporting-regressions.rst`와 함께 확인하십시오.
stable/longterm 커널 문제라도 6.0, 6.1-rc1, 6.1-rc6 같은 최신 mainline으로 확인해야 하는 이유도 reporting 문서에 설명되어 있습니다. Segment 2 뒤 예비 보고를 먼저 보내면 이미 알려진 회귀나 culprit인지 확인할 수 있습니다. 절차상 문제나 개선 제안은 커널 개발자에게 알려 주십시오.
Preparations: set up everything to build your own kernels
274-532이 절은 뒤의 모든 작업을 위한 기반을 마련합니다. 안내는 빌드와 시험을 같은 시스템에서 한다고 가정하며, 다른 시스템에서 빌드하려면 아래의 `Build kernels on a different machine` 절을 따릅니다.
예상 밖의 문제가 생길 가능성에 대비해 새 backup을 만들고 복구 도구를 준비합니다. DKMS, openZFS, VirtualBox, Nvidia graphics driver와 GPL 커널 모듈처럼 외부에서 개발한 드라이버에 의존하거나 이를 자동 빌드하는 소프트웨어를 제거합니다.
Secure Boot 같은 기술을 쓰는 플랫폼은 직접 빌드한 커널의 부팅을 허용하도록 준비합니다. 일반 x86에서는 BIOS에서 기능을 끄는 것이 가장 빠르며, `mokutil --disable-validation`으로 Linux 환경의 검증 제한을 해제하는 방법도 있습니다.
조사 범위의 good과 bad 버전을 정합니다. 단순 버그 검증은 평소 쓰는 최신 커널을 good으로 봅니다. stable/longterm 6.0.13에서 후속 mainline 계열 6.1-rc7, 6.1 또는 그 기반 6.1.5로 옮긴 뒤 회귀했다면 working 커널의 기반 mainline인 6.0을 good, 첫 broken 버전을 bad로 둡니다. 6.0이 실제 good인지는 Segment 2에서 확인합니다.
mainline 6.0에서 6.1-rc1 또는 그 기반 stable 6.1.5로 옮겨 회귀했다면 마지막 working 버전을 good, 첫 broken 버전을 bad로 둡니다. 같은 stable/longterm 계열 안에서 6.0.13에서 6.0.15로 회귀했다면 그 두 버전을 그대로 good과 bad로 삼아 해당 계열 안에서 bisect합니다. 여기서 good 버전과 working 커널은 다르며, working은 마지막으로 정상 동작한 실제 커널입니다.
working 커널로 부팅해 문제가 된 기능이 아직 정상인지 짧게 확인합니다. 빌드에는 보통 home directory에 15GB가 필요합니다. 공간이 부족하면 얕은 소스 clone과 debug symbol 비활성화를 사용해 약 4GB로 줄일 수 있습니다.
필요한 도구는 보통 `bc`, `binutils`의 `ld`, `bison`, `flex`, `gcc`, `git`, `openssl`, `pahole`, `perl`, 그리고 `libelf`와 `openssl` 개발 header입니다. 배포판별 설치 명령은 참조 절에 있습니다.
mainline 전체 clone은 2024년 초 기준 약 2.75GB를 받습니다. 불안정한 연결에는 bundle, 공간 절약에는 500MB 미만을 받는 shallow clone 대안이 있습니다. 이후 명령은 모두 `~/linux/`에서 실행합니다.
git clone -o mainline --no-checkout \
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
cd ~/linux/
git remote add -t master stable \
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.gitgood 또는 bad가 6.1.5 같은 stable/longterm 릴리스라면 해당 `linux-6.1.y` branch도 받습니다.
git remote set-branches --add stable linux-6.1.y
git fetch stable계속 working 커널을 실행 중인지 `uname -r`로 확인한 뒤, good 버전의 tag 앞에 `v`를 붙여 source를 checkout합니다.
git switch --discard-changes --detach v6.0실행 중인 커널의 build configuration을 기반으로 `.config`를 만듭니다.
make olddefconfig출력의 `# using defaults found in` 뒤가 현재 working kernelrelease가 포함된 `/boot/` 파일인지 확인합니다. `arch/x86/configs/x86_64_defconfig`라면 실행 중 커널의 구성을 찾지 못한 것이므로 참조 절에 따라 직접 배치합니다. 대신 `# configuration written to .config`만 보인다면 stale `.config`가 있는 것이므로 의도한 파일이 아니면 지우고 `make olddefconfig`를 다시 실행합니다.
빌드 시간을 크게 줄이려면 현재 장비에 불필요해 보이는 모듈을 선택적으로 끕니다. 이미 하드웨어에 맞춘 `.config`라면 이 단계는 생략합니다. 외부 USB 장치와 token을 연결하고 VM, VPN, 문제 기능을 한 번 사용한 뒤 실행합니다.
yes '' | make localmodconfig`localmodconfig`는 부팅 뒤 아직 쓰지 않은 외부 장치, virtualization, VPN 등의 모듈까지 불필요하다고 판단할 수 있습니다. 빠른 시험에서는 부팅과 문제 기능 검사가 가능하면 대개 충분하지만, 나중에 오동작하면 이 trimming을 먼저 의심하십시오. 참조 절의 방법으로 위험을 더 줄일 수 있습니다.
직접 만든 모든 커널을 구분할 수 있도록 tag와 고유 version 식별자를 설정합니다.
./scripts/config --set-str CONFIG_LOCALVERSION '-local'
./scripts/config -e CONFIG_LOCALVERSION_AUTOpanic, Oops, warning, BUG의 stack trace를 해독할 가능성이 있으면 debug symbol을 켜는 편이 좋습니다.
./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
-e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS저장 공간이 매우 부족하면 대신 debug symbol을 비활성화합니다.
./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
-d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONEDebian은 참조 절의 추가 조정으로 알려진 빌드 문제를 피해야 합니다. 다른 설정을 바꾸려면 `menuconfig`나 `nconfig`에 ncurses 개발 파일, `xconfig`에 Qt5 또는 Qt6 header가 필요합니다.
최종 조정 뒤 `.config`를 다시 처리해 안전한 위치에 보관합니다.
make olddefconfig
cp .config ~/kernel-config-workingSegment 1: try to reproduce the problem with the latest codebase
533-707이 Segment는 개발자가 현재 지원하는 코드에서도 문제가 발생하는지 확인합니다. 회귀 조사에서는 `.config` 변경이 원인인지도 가려내므로, 구성 변경 때문에 생긴 문제를 잘못 보고하는 일을 막습니다.
good과 bad가 같은 stable/longterm 계열이면 kernel.org 첫 화면에서 해당 계열이 `[EOL]` 없이 지원되는지 확인하고 최신 branch를 checkout합니다.
cd ~/linux/
git switch --discard-changes --detach stable/linux-6.1.y계열이 목록에 없거나 end of life라면 후속 stable 계열이나 mainline에서 문제가 해결됐는지 확인하십시오. 그 밖의 모든 경우에는 최신 mainline을 checkout합니다.
cd ~/linux/
git switch --discard-changes --detach mainline/master저장해 둔 구성으로 첫 커널 image와 module을 빌드합니다.
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)deb, rpm, tar package를 만들고 싶다면 참조 절의 대체 make target을 사용하며, 설치 절차도 그 package 형식에 맞춰야 합니다.
설치 전에 새 커널을 담을 공간이 남았는지 확인합니다.
df -h /boot/ /lib/modules/우선 `/boot/` 150MB와 `/lib/modules/` 200MB면 충분하다고 가정하고, 뒤에서 실제 사용량을 측정합니다. 배포판 커널과 나란히 module과 image를 설치합니다.
sudo make modules_install
command -v installkernel && sudo make install두 번째 명령은 이상적으로 image를 `/boot/`에 복사하고 initramfs를 만들며 boot loader 항목까지 추가합니다. 그러나 Arch Linux 계열과 많은 immutable 배포판은 이 작업을 전혀 하지 않거나 일부만 하므로, 빠진 단계를 배포판 문서와 참조 절에 따라 직접 수행해야 합니다. Segment 2와 3에서도 `command -v installkernel [...]` 뒤에 반복할 수 있게 추가 명령을 기록해 두십시오.
이후에도 계속 진행한다면 kernel, module, initramfs가 차지한 실제 공간을 측정합니다.
du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
du -sh /lib/modules/$(make -s kernelrelease)/두 값을 기록해 두면 bisect 중 저장 공간 부족을 예방할 수 있습니다. 방금 빌드한 커널의 kernelrelease 식별자도 표시하고 `~/kernels-built`에 누적합니다.
make -s kernelrelease | tee -a ~/kernels-built식별자를 기억해 boot menu에서 올바른 커널을 선택하고 재부팅합니다. 부팅한 커널이 방금 만든 것인지 두 출력이 같은지 확인합니다.
tail -n 1 ~/kernels-built
uname -r커널이 자신을 tainted로 표시했는지 확인합니다.
cat /proc/sys/kernel/tainted결과가 `0`이 아니면 시험을 방해할 수 있으므로 참조 절과 `Documentation/admin-guide/tainted-kernels.rst`에서 원인을 조사합니다. 새 커널에서 버그를 재현하고, 재현되지 않으면 참조 절에 따라 시험 과정의 문제를 확인합니다.
stable/longterm 커널을 빌드해 회귀가 재현됐다면 어느 개발자에게 보고할지 결정하기 위해 최신 mainline도 시험합니다. 먼저 mainline을 checkout합니다.
cd ~/linux/
git switch --discard-changes --detach mainline/master같은 구성으로 빌드·설치하고 kernelrelease를 기록한 뒤 재부팅합니다.
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot의도한 커널인지와 taint 상태를 확인합니다.
tail -n 1 ~/kernels-built
uname -r
cat /proc/sys/kernel/taintedmainline에서도 문제가 생기면 primary 개발자에게, mainline에서는 정상이고 특정 stable 계열에서만 생기면 stable team에 보고합니다. 자세한 대상 선택은 `Documentation/admin-guide/reporting-issues.rst`를 따릅니다.
현재 지원되는 코드에 문제가 있는지만 검증하려던 경우 여기서 끝입니다. 나중에 만든 커널을 제거하려면 cleanup 절을 사용합니다. 회귀를 조사 중이라면 적어도 다음 Segment까지 계속합니다.
Segment 2: check if the kernels you build work fine
708-751회귀 조사에서는 앞서 줄인 `.config`가 기대대로 작동하는지 확인해야 합니다. 잘못된 구성으로 bisect하면 전체 작업이 헛수고가 됩니다.
앞서 good으로 정한 버전, 여기서는 6.0의 source를 checkout합니다.
cd ~/linux/
git switch --discard-changes --detach v6.0이전 절과 같은 명령으로 구성·빌드·설치하고 재부팅합니다.
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot부팅 뒤 방금 만든 커널이 실행 중인지 다시 확인합니다.
tail -n 1 ~/kernels-built
uname -r이 커널에서 회귀한 기능이 정상인지 시험합니다. 정상이 아니면 참조 절의 원인 점검을 끝내기 전에는 bisect로 넘어가지 마십시오.
Segment 3: perform the bisection and validate the result
752-919준비와 예방적 빌드를 마쳤으므로 bisect를 시작할 수 있습니다. 6.0.13에서 6.1.5처럼 새 계열로 옮길 때 생긴 회귀는 보통 약 15개의 커널을 빌드합니다. 앞서 구성 범위를 줄였기 때문에 일반 x86 시스템에서는 커널 하나당 평균 10~15분 정도인 경우가 많습니다.
Git에 good 6.0과 bad 6.1.5를 알려 bisect를 시작합니다.
cd ~/linux/
git bisect start
git bisect good v6.0
git bisect bad v6.1.5Git이 checkout한 지점의 코드로 커널을 빌드·설치·부팅합니다.
cp ~/kernel-config-working .config
make olddefconfig
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
rebootcompile이 실패하면 `git bisect skip`을 실행하고 명령 묶음을 처음부터 다시 수행합니다. 최신 코드 시험을 생략했다면 저장 공간 확인과 kernelrelease 기록이 왜 필요한지 그 절을 먼저 읽으십시오.
bisect 중에는 `6.0-rc1-local-gcafec0cacaca0`처럼 이상해 보이는 release identifier가 나올 수 있지만 정상입니다. subsystem 변경이 이전 rc를 기반으로 준비된 뒤 다음 릴리스에 merge되는 Git 역사에서 비롯됩니다.
부팅한 커널이 방금 만든 것인지 확인하고 회귀 기능을 시험합니다.
cd ~/linux/
tail -n 1 ~/kernels-built
uname -r정상이라면 good으로 표시합니다.
git bisect good문제가 재현되면 bad로 표시합니다.
git bisect bad판정은 정확해야 합니다. 단 한 번의 오판도 이후 bisect를 완전히 잘못된 방향으로 보냅니다. Git은 `Bisecting: 675 revisions left to test after this (roughly 10 steps)` 같은 메시지로 남은 예상 횟수를 알리고 다음 지점을 checkout합니다. 이전 단계의 빌드와 이 단계의 판정을 반복합니다.
`cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit` 같은 메시지가 나오면 끝입니다. 이어서 culprit의 patch 설명이 길게 출력될 수 있으므로 위로 스크롤하거나 `git bisect log > ~/bisection-log`로 확인합니다.
source를 bisect 전 상태로 돌리기 전에 log와 현재 `.config`를 안전하게 보관합니다.
cd ~/linux/
git bisect log > ~/bisection-log
cp .config ~/bisection-config-culprit
git bisect reset가능하면 최신 코드에서 culprit를 revert해 회귀가 사라지는지 검증합니다. merge commit이 culprit이거나 다른 변경이 의존하면 어렵거나 불가능할 수 있지만, 성공하면 bisect가 빗나가지 않았는지 확인하고 개발자가 빠른 revert로 해결할 수 있는지도 보여 줍니다.
mainline에는 없고 특정 stable/longterm 계열에만 있는 회귀라면 영향받는 stable branch의 최신 코드를 checkout합니다.
git fetch stable
git switch --discard-changes --detach linux-6.0.y그 밖의 경우는 최신 mainline을 사용합니다.
git fetch mainline
git switch --discard-changes --detach mainline/masterstable 계열을 bisect했지만 mainline에도 문제가 있다면 `git show abcdcafecabcd`로 culprit 설명을 열어 `commit cafec0cacaca0 upstream.` 또는 `Upstream commit cafec0cacaca0`에 적힌 mainline commit id를 찾습니다. 다음 revert에는 stable backport id가 아니라 이 id를 사용합니다.
git revert --no-edit cafec0cacaca0revert가 실패하면 이 검증을 포기하고 다음 단계로 갑니다. 성공하면 다른 커널을 덮어쓰지 않고 식별할 수 있도록 tag를 바꿉니다.
cp ~/kernel-config-working .config
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'기본 `.config`는 이미 복사했으므로 다시 복사하지 않고 익숙한 순서로 빌드합니다.
make olddefconfig &&
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot마지막으로 회귀를 일으킨 기능을 시험합니다. 모든 과정이 맞았다면 revert 커널에서는 회귀가 나타나지 않아야 합니다.
Complementary tasks: cleanup during and after the bisection
920-983부팅 메뉴가 복잡해지거나 공간이 부족해질 수 있으므로 과정 중이나 후에 설치한 커널을 제거해야 할 수 있습니다. `~/kernels-built`의 기록을 보거나 다음 명령으로 kernelrelease를 빌드 순서대로 찾습니다.
ls -ltr /lib/modules/*-local*실제 bisect 중 만든 오래된 커널부터 지우는 것이 보통 좋습니다. 최신 코드와 good 버전으로 미리 만든 두 커널은 나중에 재검증할 수 있으므로 공간이 극도로 부족하지 않으면 남겨 둡니다.
`6.0-rc1-local-gcafec0cacaca0` 커널의 module directory부터 제거합니다.
sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0그 다음 배포판의 kernel-install 제거 절차를 시도합니다.
sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0많은 배포판에서는 나머지 커널 파일과 boot menu 항목도 함께 지우지만, `kernel-install`이 없거나 일부를 남기는 배포판도 있습니다. 그 경우 참조 절에 따라 직접 제거합니다.
bisect가 끝나도 즉시 모든 것을 지우지 마십시오. 최신 코드에서 회귀가 재현되고 culprit revert로 해결됐다면 그 두 커널만 당분간 보관하고 다른 `-local` 커널은 지워도 됩니다.
bisect가 merge commit에서 끝났거나 결과가 의심스러우면 재검사 요청에 대비해 가능한 많은 커널을 며칠 보관합니다. 그 밖에는 최신 코드 커널, good 버전 커널, 실제 bisect의 마지막 3~4개 커널을 한동안 남기는 것이 좋습니다.
Optional: test reverts, patches, or later versions
984-1059버그를 보고하는 동안이나 이후에 revert, debug patch, 제안된 수정안, 다른 버전을 시험하고 싶거나 요청받을 수 있습니다. 먼저 Git clone을 갱신하고 대상 최신 코드를 checkout합니다.
mainline 시험은 최신 변경을 fetch한 뒤 checkout합니다.
git fetch mainline
git switch --discard-changes --detach mainline/masterstable/longterm 시험은 아직 추가하지 않았다면 관심 계열, 예시의 6.2 branch를 remote에 추가합니다.
git remote set-branches --add stable linux-6.2.y그 계열의 최신 변경을 fetch하고 checkout합니다.
git fetch stable
git switch --discard-changes --detach stable/linux-6.2.y저장한 kernel build configuration을 복사합니다.
cp ~/kernel-config-working .config최신 코드만 시험한다면 바로 빌드 단계로 갑니다. 특정 변경의 revert를 시험하려면 하나 이상의 commit id를 지정합니다.
git revert --no-edit cafec0cacaca0식별과 덮어쓰기 방지를 위해 특별한 tag를 줍니다.
./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'patch 시험은 `/tmp/foobars-proposed-fix-v1.patch` 같은 파일로 저장해 적용하고, 여러 patch라면 각각 반복합니다.
git apply /tmp/foobars-proposed-fix-v1.patchpatch를 반영한 커널에도 짧고 고유한 tag를 줍니다.
./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'구성은 이미 복사했으므로 다시 복사하지 않고 빌드·설치·재부팅합니다.
make olddefconfig &&
make -j $(nproc --all)
# * Check if the free space suffices holding another kernel:
df -h /boot/ /lib/modules/
sudo make modules_install
command -v installkernel && sudo make install
make -s kernelrelease | tee -a ~/kernels-built
reboot새로 만든 커널로 부팅했는지 확인한 뒤 요청받은 시험을 수행합니다.
Conclusion
1060-1076단계별 안내는 여기까지입니다. 참조 절로도 해결되지 않은 문제, 오류, 개선 아이디어가 있다면 문서 관리자 Thorsten Leemhuis `<[email protected]>`에게 알려 주고 가능하면 Linux 문서 mailing list `[email protected]`를 CC하십시오.
이런 feedback은 더 많은 사용자가 이 절차를 성공적으로 수행하고, 이 안내를 바탕으로 한 유사 문서도 개선하는 데 꼭 필요합니다.
Reference section for the step-by-step guide
1077-1082이 참조 절은 위 단계별 안내의 거의 모든 항목에 대한 추가 정보, 대안, 주의점, 문제 해결 방법을 제공합니다.
Preparations for building your own kernels
1083-1094이 절의 준비 단계는 뒤의 모든 시험이 의존하는 기반입니다. 각 항목을 건너뛰지 말고 단계별 안내의 준비 절과 함께 사용하십시오.
Prepare for emergencies
1095-1109컴퓨터는 때때로 예상하지 못한 일을 하며, 특히 운영체제의 핵심인 커널을 바꾸는 과정에서는 더욱 그렇습니다. 문제가 생길 가능성은 낮지만 새 backup과 system repair·restore 도구를 미리 준비하십시오.
Remove anything related to externally maintained kernel modules
1110-1127외부에서 개발한 kernel module은 bisect 중 문제를 쉽게 일으킵니다. 더 중요한 이유는 이런 module을 사용하는 커널이 더 이상 'vanilla'로 간주되지 않아, 많은 kernel developer가 그 커널에서 발생한 회귀 보고를 다루지 않기 때문입니다. 자세한 기준은 `Documentation/admin-guide/reporting-issues.rst`에 있습니다.
Deal with techniques like Secure Boot
1128-1164많은 현대 시스템은 승인된 운영체제만 시작하게 하므로, 직접 빌드한 커널은 기본적으로 거부됩니다. 가장 바람직한 방법은 certificate로 플랫폼이 직접 빌드한 커널을 신뢰하게 만드는 것입니다. 절차는 길기 때문에 여기서는 다루지 않으며 `Documentation/admin-guide/module-signing.rst`와 관련 문서를 참고합니다.
다른 방법은 Secure Boot 같은 기능을 일시적으로 끄는 것입니다. 일반 x86에서는 BIOS Setup에서 가능하지만 기기마다 절차가 크게 다릅니다.
주류 x86 Linux 배포판에서는 `mokutil --disable-validation`으로 Linux 환경의 Secure Boot 제한을 모두 해제할 수도 있습니다. 명령이 요구하는 일회용 password를 적어 두고 재부팅합니다.
BIOS self-test 직후 Shim이 파란 화면에서 `Press any key to perform MOK management`를 보이면 countdown이 끝나기 전에 키를 누릅니다. `Change Secure Boot state`를 선택하고 MokManager가 임의로 고른 일회용 password의 세 문자를 입력한 뒤 validation 비활성화를 확인하고 재부팅을 허용합니다.
Boot the last kernel that was working
1165-1177마지막 working 커널로 부팅해 회귀한 기능이 실제로 정상인지 다시 짧게 확인합니다. 그래야 뒤의 configuration 생성과 trimming이 올바른 커널 상태를 기준으로 작동합니다.
Space requirements
1178-1195안내의 저장 공간 수치는 여유를 크게 잡은 대략적인 값이므로 실제로는 더 적게 필요할 수 있습니다. 공간이 제한되면 debug symbol 단계와 참조 설명을 반드시 확인하십시오. 이를 끄면 사용량을 수 GB 줄일 수 있습니다.
Bisection range
1196-1220검사할 commit 범위는 대체로 간단하지만, 한 stable 계열의 릴리스에서 다음 계열의 릴리스로 옮길 때 회귀했다면 Git 역사에 직선적인 조상 관계가 없어 주의가 필요합니다.
예를 들어 mainline은 6.0 뒤 6.1로 진행하지만 stable 6.0.y는 옆으로 분기합니다. 이론상 6.0.13에서만 정상인 이유가 6.0.y에 들어갔지만 mainline이나 6.1.y에는 없는 수정 때문일 수 있습니다.
stable/longterm 유지 방식상 이런 경우는 보통 없어 6.0을 good으로 가정해도 안전합니다. Segment 2에서 직접 빌드해 이 가정을 확인하며, 6.0.13과 6.1.5 사이를 바로 bisect하려 해도 Git이 결국 같은 확인을 요구합니다.
Install build requirements
1221-1271커널은 상당히 독립적이지만 compiler 외에 구성에 따라 library가 더 필요합니다. 배포판과 만들 커널의 configuration에 따라 설치 방법이 달라집니다. 다음은 주류 배포판의 일반적인 예입니다.
Arch Linux와 파생 배포판:
sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \
pahole perl zlib ncurses qt6-baseDebian, Ubuntu와 파생 배포판:
sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \
libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \
libncurses-dev qt6-base-dev g++Fedora와 파생 배포판:
sudo dnf install binutils \
/usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \
/usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction}openSUSE와 파생 배포판:
sudo zypper install bc binutils bison dwarves flex gcc git \
kernel-install-tools libelf-devel make modutils openssl openssl-devel \
perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel이 명령들은 자주 필요하지만 항상 필요한 것은 아닌 package도 설치합니다. `menuconfig`나 `nconfig`로 설정을 바꾸지 않는다면 ncurses 개발 header를, `xconfig`를 쓰지 않는다면 Qt6 header를 생략할 수 있습니다.
이 안내에서 다루지 않는 작업, 예를 들어 커널 `tools/` directory의 utility를 빌드하려면 추가 library와 개발 header가 필요할 수 있습니다.
Download the sources using Git
1272-1316단계별 안내는 Linus의 mainline repository를 full Git clone하는 방법을 사용합니다. 연결이 불안정하면 Git bundle, 전체 repository 다운로드가 너무 오래 걸리거나 공간이 부족하면 shallow clone을 선택할 수 있습니다.
Linux mainline source를 bundle로 받을 때는 다음 명령을 사용합니다. `wget`이 실패하면 같은 명령을 다시 실행하면 중단된 위치에서 이어집니다.
wget -c \
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
git clone --no-checkout clone.bundle ~/linux/
cd ~/linux/
git remote remove origin
git remote add mainline \
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git fetch mainline
git remote add -t master stable \
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.gitDownloading Linux mainline sources using a shallow clone
1317-1362최신 mainline 코드만 얕게 받으려면 다음 명령을 실행합니다.
git clone -o mainline --no-checkout --depth 1 -b master \
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
cd ~/linux/
git remote add -t master stable \
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git그다음 good 버전의 mainline 릴리스에서 두 번째 선행 릴리스까지 역사를 깊게 합니다. good이 6.0 또는 6.0.13이면 첫 선행은 5.19, 두 번째는 5.18이므로 다음처럼 받습니다.
git fetch --shallow-exclude=v5.18 mainline이후 단계별 안내처럼 stable repository를 remote로 추가하고 필요한 stable branch를 모두 더합니다. shallow clone은 몇 가지 특성이 있습니다.
bisect에는 겉보기보다 몇 mainline 버전 더 깊은 역사가 필요합니다. 6.1..6.2 범위의 commit도 내부적으로 6.0-rc2나 5.19-rc3 같은 더 이른 release를 기반으로 할 수 있어, 역사가 얕으면 Git이 대부분의 commit을 revert하거나 describe하지 못합니다.
이 문서는 필요한 가장 이른 Git tag를 `git fetch --shallow-exclude=`로 지정합니다. 대신 `--shallow-since=`에 `'2023-07-15'` 같은 절대 날짜나 `'12 months'` 같은 상대 날짜를 줄 수 있습니다. mainline bisect라면 good 커널 기반 mainline 릴리스보다 최소 7개월 전까지 역사를 확보하십시오.
역사를 깊게 할 때 `fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da`가 나오면 `git repack -d`를 실행한 뒤 다시 시도합니다.
Start defining the build configuration for your kernel
1363-1422이 단계부터 build artifact를 만들거나 바꿉니다. 안내는 단순화를 위해 source tree 안에 저장합니다. 별도로 저장하려면 `~/linux-builddir/` 같은 directory를 만들고 모든 make 호출에 `O=~/linux-builddir/`를 추가합니다. `./scripts/config [...]`에는 `--file ~/linux-builddir/.config`도 지정해야 합니다.
구성 생성에는 두 가지 흔한 문제가 있습니다. 첫째, build directory에 기존 `.config`, 예를 들어 `~/linux/.config`가 있으면 oldconfig target이 그것을 사용합니다. 의도한 경우가 아니면 삭제해야 하며, 절차를 되돌아와 처음부터 다시 구성할 때 특히 중요합니다.
둘째, olddefconfig가 실행 중 커널의 `.config`를 찾지 못해 default를 사용할 수 있습니다. 배포판이 설정을 제공하는 위치를 찾아 `~/linux/.config`에 직접 놓으십시오. `/proc/config.gz`가 있다면 다음처럼 만듭니다.
zcat /proc/config.gz > .config그 뒤 `make olddefconfig`를 다시 실행해 현재 source 요구에 맞춥니다. olddefconfig는 정의되지 않은 option을 default로 설정합니다. 직접 답하려면 `make oldconfig`를 쓰고 확신이 없을 때 Enter로 default를 선택합니다. bisect에서는 새 보안 기능 등을 뜻하지 않게 켜 회귀처럼 보이는 문제를 만들지 않도록 default가 보통 낫습니다.
한 커널용 config, 예를 들어 6.1의 파일을 훨씬 오래된 5.15에 쓰면 이상한 일이 생길 수 있습니다. 그러므로 working 커널의 config를 사용해야 하며 broken 커널의 것을 넣지 마십시오.
다른 장비용 커널을 빌드한다면 그 장비에서 보통 `ls /boot/config-$(uname -r)`로 configuration을 찾고 빌드 장비의 `~/linux/.config`에 복사한 뒤 `make olddefconfig`를 실행합니다.
Trim the build configuration for your kernel
1423-1471`localmodconfig`는 실행하기 전 한 번도 수행하지 않은 작업에 필요한 module을 쉽게 끌 수 있습니다. 어떤 module은 VM, VPN, loop-mounted CD/DVD ISO, CIFS·NFS network share, 외부 장치, btrfs·ext4·FAT·NTFS·XFS 같은 filesystem을 실제로 사용할 때 처음 자동 load되기 때문입니다.
2FA key, headset, webcam, 외부 저장 장치를 연결하고 평소 작업을 가능한 한 수행해 위험을 줄일 수 있지만 모든 경우를 떠올리기는 어렵습니다. 빠른 시험에서는 보통 핵심 기능이 남으므로 지나치게 걱정하지 말고, 빠진 기능은 나중에 다시 켜 재빌드할 수 있습니다.
직접 빌드한 커널을 계속 쓸 계획이라면 `modprobed-db`가 몇 주 동안 실제 사용 module을 기록하게 한 뒤 `LSMOD=<path>`로 그 목록을 localmodconfig에 전달할 수 있습니다.
yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig다른 장비용 trimmed kernel은 그 장비에서 `lsmod > lsmod_foo-machine`을 실행해 파일을 빌드 host로 복사한 뒤 다음처럼 사용합니다.
yes '' | make LSMOD=~/lsmod_foo-machine localmodconfigTag the kernels about to be build
1472-1487특별한 tag와 고유 version identifier는 배포판 커널과 이 과정에서 만든 커널을 구분합니다. 파일과 directory 이름의 `-local`로 알아볼 수 있고, boot menu에서 올바른 항목을 고르며 bisect 중 혼란스러운 version number를 추적하는 데도 도움이 됩니다.
Decide to enable or disable debug symbols
1488-1516kernel panic, Oops, warning, BUG가 발생했을 때 debug symbol이 있으면 코드의 정확한 실패 지점을 찾을 수 있습니다. 그러나 정보 수집과 포함에는 시간과 공간이 듭니다.
2022년 말 기준 localmodconfig로 줄인 일반적인 x86 kernel의 build artifact는 debug symbol을 켜면 약 5GB, 끄면 1GB 미만이었습니다. kernel image와 module도 커져 `/boot/` 공간과 load 시간이 늘어납니다.
stack trace를 해독할 가능성이 낮고 작은 커널이 필요하면 끄고, 나중에 필요해지면 켜서 다시 빌드하십시오. 해독할 가능성이 충분하면 반드시 켜야 합니다. 자세한 과정은 `Documentation/admin-guide/reporting-issues.rst`의 `Decode failure messages` 절에 있습니다.
Adjust build configuration
1517-1569필요에 따라 이 시점에서 다른 kernel configuration option을 조정할 수 있습니다. 일부 일반 배포판은 알려진 빌드 문제를 피하기 위한 배포판별 조정이 필요합니다.
Debian에서는 build를 실패하게 할 오래된 certificate file 참조를 제거합니다.
./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''대신 Debian handbook 설명처럼 필요한 certificate를 받아 option이 가리키게 하거나 `Documentation/admin-guide/module-signing.rst`에 따라 직접 만들 수도 있습니다.
개별 조정은 `make menuconfig` 또는 `make nconfig`의 text UI로 기능을 켜고 끌 수 있고, graphical utility는 `make xconfig`를 사용합니다. 각각 ncurses 또는 Qt5·Qt6 개발 library가 필요하며, 빠진 항목은 error message에 나타납니다.
Put the .config file aside
1570-1586준비한 `.config`를 별도 위치에 보관하고, 이 안내에서 다른 커널을 빌드할 때마다 build directory로 다시 복사합니다. 서로 다른 version을 오가면 `.config`가 예상 밖으로 바뀔 수 있고, 그 부작용이 시험을 혼란스럽게 하거나 bisect 결과를 무의미하게 만들 수 있습니다.
Try to reproduce the problem with the latest codebase
1587-1640이미 배포판 커널로 확인했거나 같은 stable/longterm 계열 안의 회귀라도 최신 codebase를 시험해야 합니다. 이 단계에서 설정 문제를 미리 만나면 실제 bisect 중 source 지점의 무관한 build·boot 실패와 자신의 환경 문제를 구분하기 쉽습니다.
working과 broken 커널 사이의 build configuration 변경이 원인인지도 배제합니다. 새 커널에서 배포판이 보안 기능을 켰고 그 기능이 사용자의 작업을 막았다면 upstream 관점에서는 커널 회귀가 아닐 수 있으므로, 이 경우 bisect는 시간 낭비입니다. 자세한 기준은 `Documentation/admin-guide/reporting-regressions.rst`에 있습니다.
최신 mainline에서 이미 고쳐졌다면 bisect할 필요가 없습니다. stable/longterm 회귀도 mainline 변경의 backport에서 생기는 경우가 많아 mainline에서 먼저 수정되어야 하며, fix가 이미 backport되는 중일 수 있습니다.
stable/longterm 계열 안의 회귀는 그 계열만의 문제인지 mainline에도 있는지 알아야 보고 대상이 달라집니다. stable 전용 회귀는 stable team 책임이고, mainline에도 있는 회귀는 일반 Linux 개발자와 maintainer가 다루며 stable team에는 fix가 준비된 뒤 backport만 요청하면 됩니다. 잘못된 대상에게 보내면 무시되거나 이 구분부터 다시 요구받을 수 있습니다.
Check out the latest Linux codebase
1641-1654더 최신 codebase가 문제를 고쳤는지 나중에 다시 확인하려면 앞에서 사용한 `git fetch --shallow-exclude [...]`를 다시 실행해 local Git repository를 갱신한 뒤 checkout합니다.
Build your kernel
1655-1730준비한 config로 첫 kernel image와 module을 빌드합니다. 이 단계에서는 여러 문제가 생길 수 있으므로 아래 방법으로 원인을 좁힙니다. 직접 deb, rpm, tar package를 만드는 대안도 있습니다.
build error는 장비 설정 때문에 빨리 고칠 수 있는 경우도 있고, code 자체의 문제여서 개발자가 고쳐야 하는 경우도 있습니다. 상세 실패 메시지를 보고 조사하기 위해 verbose mode로 병렬성을 끈 채 다시 빌드합니다.
make V=1`V=1`은 실제 error를 볼 수 있게 상세 출력을 켭니다. `-j $(nproc --all)`을 생략하면 CPU core 병렬 출력이 뒤섞이는 것도 막습니다. 잠시 뒤 같은 error가 다시 나면 가장 핵심적인 한 줄을 찾습니다.
그 줄에서 일반적이지 않은 핵심 4~8단어를 골라 검색하고 username이나 `/home/username/linux/` 같은 local path는 제거합니다. 일반 검색 engine을 먼저 쓰고 `lore.kernel.org/all/`의 Linux kernel mailing list도 검색합니다. 대부분 이미 누군가 겪은 문제라 설명, solution, patch, workaround를 찾을 수 있습니다. 없으면 검색어와 error line을 바꿔 반복합니다.
단계별 안내는 x86의 `bzImage`, `modules` 같은 기본 make target으로 빌드한 뒤 설치합니다. 대신 `make -j $(nproc --all) bindeb-pkg`는 deb, `make -j $(nproc --all) binrpm-pkg`는 rpm, `make -j $(nproc --all) tarbz2-pkg`는 bz2 tarball을 만듭니다. 다른 target은 `make help`에서 찾을 수 있고, 이미 일반 빌드를 했다면 결과를 재사용합니다.
deb나 rpm target을 사용하면 안내의 수동 설치·제거 절차를 무시하고 dpkg, rpm 또는 apt, aptitude, dnf/yum, zypper 같은 package 관리 도구를 사용합니다. 생성된 package는 여러 배포판에서 작동하도록 설계되어 해당 배포판의 공식 kernel package와 다르게 동작할 수 있습니다.
Put the kernel in place
1731-1773설치 명령 뒤 해야 할 일은 배포판의 `/sbin/installkernel` 존재 여부와 구현에 달려 있습니다. 파일이 있으면 kernel build system이 실제 image 설치를 위임합니다.
거의 모든 배포판에서 image는 보통 `/boot/vmlinuz-<kernelrelease_id>`에 놓이고 `System.map-<kernelrelease_id>`가 함께 생깁니다. 대부분 initramfs 또는 initrd도 `/boot/initramfs-<kernelrelease_id>.img`나 `/boot/initrd-<kernelrelease_id>`로 만듭니다.
일반 배포판은 initramfs로 부팅하므로 반드시 먼저 `modules_install` target을 실행해야 generator가 필요한 module을 찾습니다. 일부 installkernel은 bootloader configuration 항목도 추가합니다.
installkernel이 없거나 일부만 처리하면 배포판 문서에 따라 나머지를 직접 수행합니다. 확실하지 않다면 image와 System.map을 다음처럼 설치합니다.
sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)그다음 배포판 도구로 initramfs를 만들고 bootloader configuration에 커널을 추가한 뒤 재부팅합니다.
Storage requirements per kernel
1774-1794bisect 중 만든 커널은 특히 debug symbol을 켰을 때 `/boot/`와 `/lib/modules/` 공간을 많이 씁니다. volume이 가득 차면 이전에 정상 부팅하던 커널까지 실패할 수 있으므로 커널 하나의 실제 사용량을 알아야 합니다.
안내의 `/boot/*$(make -s kernelrelease)*` pattern은 보통 필요한 파일을 모두 찾지만 path와 이름 규칙이 강제된 것은 아닙니다. 배포판에 따라 다른 위치도 확인해야 합니다.
Check if your newly built kernel considers itself 'tainted'
1795-1813Linux는 이후에 전혀 무관해 보이는 error를 일으킬 수 있는 일이 생기면 자신을 tainted로 표시합니다. 보고한 bug가 생긴 바로 그때 flag가 설정된 경우가 아니라면 개발자는 tainted kernel 보고를 무시하거나 제한적으로 대응할 수 있습니다.
`Documentation/admin-guide/tainted-kernels.rst`에 따라 원인을 확인하십시오. 그렇지 않으면 자신의 시험 자체가 잘못될 수도 있습니다.
Check the kernel built from a recent mainline codebase
1814-1839최신 codebase로 만든 커널에서 버그나 회귀가 나타나지 않는 흔한 이유는 그 사이 수정됐거나, 커널 제공자가 build configuration을 바꾼 것이 원인이었기 때문입니다.
race condition이라 trimmed configuration, debug symbol 설정, compiler 등 차이로 재현되지 않을 수도 있습니다. stable/longterm 커널에서 만난 회귀라면 특정 계열에만 있는 문제일 수 있으므로 다음 단계에서 확인합니다.
Check the kernel built from the latest stable/longterm codebase
1840-1854stable/longterm 계열 안의 회귀가 최신 mainline에서는 재현되지 않는다면, 해당 stable 계열의 최신 codebase가 이미 문제를 고쳤는지 확인합니다. 이 커널에서도 회귀가 없다면 bisect할 필요가 없을 가능성이 큽니다.
Ensure the 'good' version is really working well
1855-1884이 절은 알려진 working 기준을 다시 세웁니다. 생략하고 싶을 수 있지만, 준비한 `.config`가 정말 예상대로 작동하는지 확인하지 않으면 10개 이상의 커널을 쓸데없이 시험한 뒤에야 구성 문제를 의심하게 됩니다.
많은 사용자는 patch나 add-on module이 들어간 비-vanilla 커널을 평소 사용합니다. 따라서 회귀한 기능이 good 버전의 vanilla build에서는 애초에 작동하지 않았을 수도 있습니다.
서로 다른 stable/longterm 계열, 예를 들어 6.0.13..6.1.5 사이의 회귀라면 앞에서 good으로 가정한 6.0이 실제로 정상인지도 이 단계가 확인합니다.
Build your own version of the 'good' kernel
1885-1916첫 self-built good 커널에서 기능이 작동하지 않으면 원인을 해결한 뒤 진행합니다. 먼저 taint 상태와 `dmesg`를 보고 무관한 실패가 있었는지 확인합니다.
localmodconfig가 필요한 module을 껐을 수 있습니다. 마지막 working 커널의 config로 다시 만들고 trimming을 생략하거나, build 시간을 줄이기 위해 일부 기능만 수동으로 끄십시오.
커널 회귀가 아니라 우연한 일시적 문제, broken initramfs 또는 initrd, 새 firmware file, 갱신된 userland software가 원인일 수도 있습니다. 배포판 커널이 추가한 기능이라 당시 vanilla Linux에는 없었을 가능성도 있습니다.
`.config` 문제를 고쳤다면 최신 codebase 커널도 새 구성으로 다시 빌드해야 합니다. 이전 mainline 및 stable/longterm 최신판 시험은 잘못된 구성을 사용했을 가능성이 높습니다.
Perform a bisection and validate the result
1917-1928모든 준비와 예방적 빌드를 마쳤으므로 bisect를 시작할 수 있습니다. 이 Segment의 단계는 bisect 수행과 결과 검증을 담당합니다.
Start the bisection
1929-1942Git에 good과 bad version을 알려 bisect를 시작하면 마지막 명령이 두 변경 사이의 대략 중간 commit을 자동 checkout해 시험할 준비를 합니다.
Build a kernel from the bisection point
1943-1970Git이 checkout한 코드로 앞과 같은 명령을 사용해 커널을 빌드·설치·부팅합니다. 해당 지점의 무관한 code 문제로 build나 boot가 실패하면 다음 명령을 실행합니다.
git bisect skipGit은 근처의 다른 commit을 checkout하며 운이 좋으면 그 지점은 정상 동작합니다. 그 뒤 이 단계를 다시 시작합니다.
bisect 중 version identifier가 이상해 보이는 것은 subsystem 변경이 다음 mainline release, 예를 들어 6.2용으로 준비되면서 6.1-rc1이나 6.0을 기반으로 하고, 6.1 뒤에 rebase나 squash 없이 6.2로 merge되기 때문입니다.
Bisection checkpoint
1971-1984방금 만든 커널에서 회귀 기능을 시험하고 Git에 정확한 결과를 알려야 합니다. 한 번만 잘못 판정해도 이후 bisect가 모두 잘못되어 나머지 시험이 무의미해집니다.
Put the bisection log away
1985-2004커널 하나를 good 또는 bad로 잘못 선언하면 결과가 쓸모없어져 보통 처음부터 다시 해야 합니다. bisection log가 있으면 어디서 잘못됐는지 찾아 10개 이상 대신 몇 개만 다시 빌드해 바로잡을 수 있습니다.
현재 `.config`도 함께 보관합니다. 회귀를 보고한 뒤 개발자가 요청할 가능성이 충분히 있습니다.
Try reverting the culprit
2005-2023culprit를 최신 codebase에서 revert해 회귀가 사라지는지 시험하는 것은 선택 사항이지만 가능한 경우 수행하십시오. 보고 뒤 개발자가 요청할 가능성이 높고, 이미 빌드 흐름을 갖춘 상태라 한 번 더 빌드하는 부담이 크지 않습니다.
mainline에도 있는 회귀를 stable/longterm 계열로 bisect했고 mainline에서 Git revert가 실패했다면, 영향받는 stable/longterm 계열에서 culprit를 revert해 보십시오. 거기서 성공하면 그 커널을 시험합니다.
Cleanup steps during and after following this guide
2024-2035과정 중이나 뒤에 설치한 커널을 제거해야 할 수 있습니다. 다음 두 절은 bisect 중과 완료 뒤의 cleanup 절차를 설명합니다.
Cleaning up during the bisection
2036-2076이 과정에서 설치한 커널은 두 위치에만 있고 식별이 쉬워 나중에 제거하기 어렵지 않습니다. 수동 설치로 배포판 package system을 우회했더라도 정리할 수 있습니다.
첫 위치는 커널별 module을 담는 `/lib/modules/<kernelrelease>` directory입니다. 해당 커널의 module은 이 directory를 지우면 됩니다.
두 번째는 `/boot/`이며 보통 release name이 포함된 파일 2~5개가 있습니다. 정확한 파일 수와 이름은 installkernel과 initramfs generator에 따라 다릅니다. 일부 배포판의 `kernel-install remove...`는 파일과 bootloader menu 항목을 모두 지우지만, 다른 배포판에서는 직접 해야 합니다.
`6.0-rc1-local-gcafec0cacaca0`의 주요 세 파일은 대화형으로 다음처럼 제거합니다.
rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0그 뒤 같은 식별자가 들어간 다른 `/boot/` 파일도 확인하고 bootloader configuration 항목을 제거합니다. 수동 삭제에서 `*` wildcard를 조심하십시오. 6.0이나 6.0.1만 지우려다 6.0.13 파일까지 지울 수 있습니다.
Cleaning up after the bisection
2077-2105bisect가 끝나도 설정한 것을 바로 모두 제거하지 마십시오. 추가 시험에 다시 필요할 수 있습니다. 공간이 매우 부족하면 `rm -rf ~/linux/*`로 build artifact와 Linux source를 지울 수 있습니다.
이 명령은 `~/linux/.git/`을 남기므로 `git reset --hard`로 source를 되돌릴 수 있습니다. repository까지 지우는 것은 현명하지 않습니다. 개발자가 debug patch나 제안된 fix를 시험할 커널을 더 빌드해 달라고 할 가능성이 있습니다.
추가 시험은 `Optional tasks: test reverts, patches, or later versions` 절을 따르며, 같은 이유로 `~/kernel-config-working`도 몇 주 보관합니다.
Test reverts, patches, or later versions
2106-2120이 절의 명령은 대체로 직관적입니다. kernel tag를 정할 때 예시보다 훨씬 길게 만들지 마십시오. kernelrelease identifier가 63자를 넘으면 문제가 생깁니다.
Additional information
2121-2125아래에는 다른 장비에서 커널을 빌드하는 절차와 bisect를 더 깊이 공부할 자료가 있습니다.
Build kernels on a different machine
2126-2189다른 시스템에서 compile하려면 먼저 나중에 커널을 설치하고 시험할 장비에서 안내를 시작합니다. working 커널로 부팅해 문제 기능을 확인한 뒤 loaded module 목록을 `lsmod > ~/test-machine-lsmod`로 저장합니다.
같은 장비의 실행 중 커널 build configuration을 찾아 `~/test-machine-config-working`으로 저장하고, 두 파일을 build host의 home directory로 옮깁니다. 이후 충분한 공간을 확인하는 단계부터 build host에서 계속합니다.
처음 `make olddefconfig`를 실행하기 전에 test 장비의 working configuration을 복사합니다.
cp ~/test-machine-config-working ~/linux/.config불필요한 module을 끄는 단계에서는 test 장비의 module 목록을 지정합니다.
yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig이후 compile·install·reboot 지시는 build host에서 생략하고 다음처럼 gzipped tar package를 만듭니다.
cp ~/kernel-config-working .config
make olddefconfig &&
make -j $(nproc --all) targz-pkg마지막 출력 줄에 생성 파일명이 나오며, x86용 `6.0.0-rc1-local-g928a87efa423` 커널은 보통 `~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz`입니다. 이 파일을 test 장비의 home directory로 복사합니다.
test 장비에서 새 커널 공간을 확인하고 root filesystem에 압축을 풉니다.
sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C /그 뒤 initramfs를 만들고 bootloader configuration에 커널을 추가합니다. 일부 배포판은 다음 명령이 두 작업을 모두 합니다.
sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa423재부팅해 의도한 커널인지 확인합니다. 다른 architecture용 빌드도 가능하며 cross-compiler를 설치하고 모든 make 호출에 `make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]` 같은 parameter를 추가합니다.
Additional reading material
2190-2222추가 자료로 Git 문서의 `git bisect` man page와 `fighting regressions with 'git bisect'`, kernel developer Nathan Chancellor의 `Working with git bisect`, `Using Git bisect to figure out when brokenness was introduced`, LWN의 `Fully automated bisecting with 'git bisect run'`을 참고할 수 있습니다.
이 문서는 Thorsten Leemhuis `<[email protected]>`가 관리합니다. 오탈자나 작은 실수는 직접 알려도 되며, 문서 변경을 기여할 때는 저작권 처리를 위해 `[email protected]`를 CC하고 `Documentation/process/submitting-patches.rst`의 `Sign your work - the Developer's Certificate of Origin`에 따라 sign-off하십시오.
원문 상단에 명시된 대로 이 text는 GPL-2.0+ 또는 CC-BY-4.0으로 제공됩니다. CC-BY-4.0만으로 배포할 때는 저자를 `The Linux kernel development community`로 표시하고 원본 source URL을 연결합니다.
CC-BY-4.0 적용 범위는 Linux kernel source에 들어 있는 이 RST 파일의 내용뿐입니다. kernel build system 등으로 처리된 version은 더 제한적인 license의 다른 파일 내용을 포함할 수 있습니다.
요약과 해설
verify-bugs-and-bisect-regressions.rst:1-2222이 문서는 배포판 커널 사용자가 upstream에 신뢰할 수 있는 보고를 내기까지의 실전 절차입니다. 외부 module과 Secure Boot를 정리하고 working kernel의 `.config`를 줄인 뒤, 최신 mainline과 good 버전을 직접 빌드해 환경·구성 차이를 먼저 배제합니다.
명령 순서만 따르는 것보다 각 checkpoint의 의미가 중요합니다. taint, 실제 부팅한 kernelrelease, 저장 공간, good/bad 판정을 매번 확인해야 한 번의 오판이나 환경 문제로 전체 bisect가 무효가 되는 일을 막을 수 있습니다.