요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
1
=========================
2
Hardware Latency Detector
3
=========================
5
Introduction
6
-------------
8
The tracer hwlat_detector is a special purpose tracer that is used to
9
detect large system latencies induced by the behavior of certain underlying
10
hardware or firmware, independent of Linux itself. The code was developed
11
originally to detect SMIs (System Management Interrupts) on x86 systems,
12
however there is nothing x86 specific about this patchset. It was
13
originally written for use by the "RT" patch since the Real Time
14
kernel is highly latency sensitive.
16
SMIs are not serviced by the Linux kernel, which means that it does not
17
even know that they are occurring. SMIs are instead set up by BIOS code
18
and are serviced by BIOS code, usually for "critical" events such as
19
management of thermal sensors and fans. Sometimes though, SMIs are used for
20
other tasks and those tasks can spend an inordinate amount of time in the
21
handler (sometimes measured in milliseconds). Obviously this is a problem if
22
you are trying to keep event service latencies down in the microsecond range.
24
The hardware latency detector works by hogging one of the cpus for configurable
25
amounts of time (with interrupts disabled), polling the CPU Time Stamp Counter
26
for some period, then looking for gaps in the TSC data. Any gap indicates a
27
time when the polling was interrupted and since the interrupts are disabled,
28
the only thing that could do that would be an SMI or other hardware hiccup
29
(or an NMI, but those can be tracked).
31
Note that the hwlat detector should *NEVER* be used in a production environment.
32
It is intended to be run manually to determine if the hardware platform has a
33
problem with long system firmware service routines.
35
Usage
36
------
38
Write the ASCII text "hwlat" into the current_tracer file of the tracing system
39
(mounted at /sys/kernel/tracing or /sys/kernel/tracing). It is possible to
40
redefine the threshold in microseconds (us) above which latency spikes will
41
be taken into account.
43
Example::
45
# echo hwlat > /sys/kernel/tracing/current_tracer
46
# echo 100 > /sys/kernel/tracing/tracing_thresh
48
The /sys/kernel/tracing/hwlat_detector interface contains the following files:
50
- width - time period to sample with CPUs held (usecs)
51
must be less than the total window size (enforced)
52
- window - total period of sampling, width being inside (usecs)
54
By default the width is set to 500,000 and window to 1,000,000, meaning that
55
for every 1,000,000 usecs (1s) the hwlat detector will spin for 500,000 usecs
56
(0.5s). If tracing_thresh contains zero when hwlat tracer is enabled, it will
57
change to a default of 10 usecs. If any latencies that exceed the threshold is
58
observed then the data will be written to the tracing ring buffer.
60
The minimum sleep time between periods is 1 millisecond. Even if width
61
is less than 1 millisecond apart from window, to allow the system to not
62
be totally starved.
64
If tracing_thresh was zero when hwlat detector was started, it will be set
65
back to zero if another tracer is loaded. Note, the last value in
66
tracing_thresh that hwlat detector had will be saved and this value will
67
be restored in tracing_thresh if it is still zero when hwlat detector is
68
started again.
70
The following tracing directory files are used by the hwlat_detector:
72
in /sys/kernel/tracing:
74
- tracing_threshold - minimum latency value to be considered (usecs)
75
- tracing_max_latency - maximum hardware latency actually observed (usecs)
76
- tracing_cpumask - the CPUs to move the hwlat thread across
77
- hwlat_detector/width - specified amount of time to spin within window (usecs)
78
- hwlat_detector/window - amount of time between (width) runs (usecs)
79
- hwlat_detector/mode - the thread mode
81
By default, one hwlat detector's kernel thread will migrate across each CPU
82
specified in cpumask at the beginning of a new window, in a round-robin
83
fashion. This behavior can be changed by changing the thread mode,
84
the available options are:
86
- none: do not force migration
87
- round-robin: migrate across each CPU specified in cpumask [default]
88
- per-cpu: create one thread for each cpu in tracing_cpumask
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
Hardware latency detector 소개
1-34`hwlat_detector` tracer는 Linux 자체와 무관하게 특정 하위 hardware 또는 firmware 동작이 유발하는 큰 system latency를 찾는 특수 목적 tracer다. 원래 x86 system의 SMI(System Management Interrupt)를 감지하려고 개발됐지만 patchset 자체는 x86에 종속되지 않는다. Real-Time kernel은 latency에 매우 민감하므로 처음에는 RT patch에서 사용하기 위해 작성됐다.
Linux kernel은 SMI를 처리하지 않으며 SMI 발생 사실조차 알지 못한다. SMI는 BIOS code가 설정하고 처리하며, 보통 thermal sensor나 fan 관리 같은 critical event에 사용된다. 그러나 다른 작업에 사용된 SMI handler가 때로 millisecond 단위로 너무 오래 실행되면 microsecond 수준의 event service latency를 유지하려는 system에 문제가 된다.
Hardware latency detector는 interrupt를 비활성화한 채 CPU 하나를 설정 가능한 시간 동안 점유하고 CPU Time Stamp Counter를 polling한다. 이후 TSC data의 gap을 찾는다. interrupt가 꺼져 있으므로 polling을 끊을 수 있는 것은 SMI, 다른 hardware hiccup 또는 추적 가능한 NMI뿐이다.
`hwlat` detector는 production 환경에서 절대로 사용하면 안 된다. Hardware platform에 긴 system firmware service routine 문제가 있는지 수동으로 확인하기 위한 도구다.
CPU 점유→interrupt disable
TSC polling→연속 timestamp 비교
TSC gap→SMI / hardware hiccup / NMI 후보
threshold 초과→trace ring buffer 기록
Interrupt를 차단한 polling 구간의 TSC gap으로 Linux 밖에서 소비된 시간을 드러낸다.
=========================
Hardware Latency Detector
=========================
Introduction
-------------
The tracer hwlat_detector is a special purpose tracer that is used to
detect large system latencies induced by the behavior of certain underlying
hardware or firmware, independent of Linux itself. The code was developed
originally to detect SMIs (System Management Interrupts) on x86 systems,
however there is nothing x86 specific about this patchset. It was
originally written for use by the "RT" patch since the Real Time
kernel is highly latency sensitive.
SMIs are not serviced by the Linux kernel, which means that it does not
even know that they are occurring. SMIs are instead set up by BIOS code
and are serviced by BIOS code, usually for "critical" events such as
management of thermal sensors and fans. Sometimes though, SMIs are used for
other tasks and those tasks can spend an inordinate amount of time in the
handler (sometimes measured in milliseconds). Obviously this is a problem if
you are trying to keep event service latencies down in the microsecond range.
The hardware latency detector works by hogging one of the cpus for configurable
amounts of time (with interrupts disabled), polling the CPU Time Stamp Counter
for some period, then looking for gaps in the TSC data. Any gap indicates a
time when the polling was interrupted and since the interrupts are disabled,
the only thing that could do that would be an SMI or other hardware hiccup
(or an NMI, but those can be tracked).
Note that the hwlat detector should *NEVER* be used in a production environment.
It is intended to be run manually to determine if the hardware platform has a
problem with long system firmware service routines.
사용법과 sampling 주기
35-68Tracing system의 `current_tracer` 파일에 ASCII text `hwlat`를 쓰면 tracer가 시작된다. Microseconds 단위의 `tracing_thresh`를 바꾸면 어느 크기 이상의 latency spike를 고려할지 다시 정할 수 있다.
# echo hwlat > /sys/kernel/tracing/current_tracer
# echo 100 > /sys/kernel/tracing/tracing_thresh`/sys/kernel/tracing/hwlat_detector` interface의 `width`는 CPU를 점유해 sampling하는 시간이고, `window`는 width를 포함하는 전체 sampling period다. `width`는 반드시 `window`보다 작아야 한다.
파일의미단위·제약
`width`CPU를 점유해 TSC를 sampling하는 시간usecs, window보다 작음
`window`width를 포함한 전체 sampling periodusecs
`tracing_thresh`기록할 최소 latencyusecs, 0이면 시작 시 10
한 window 안에서 busy polling과 system 양보 시간을 나눈다.
기본값은 `width=500000`, `window=1000000`이다. 즉 1초마다 0.5초 동안 detector가 spin한다. `hwlat` tracer를 활성화할 때 `tracing_thresh`가 0이면 기본값 10 usecs로 바뀌며, threshold를 넘는 latency가 관측되면 trace ring buffer에 기록된다.
System이 완전히 굶지 않도록 각 period 사이에는 최소 1 millisecond를 sleep한다. `window - width`가 1 millisecond보다 작아도 이 최소 sleep은 유지된다.
Detector 시작 시 `tracing_thresh`가 0이었다면 다른 tracer를 load할 때 다시 0으로 되돌린다. Detector가 마지막으로 사용한 threshold는 저장하며, 다시 시작할 때 `tracing_thresh`가 여전히 0이면 그 값을 복원한다.
Usage
------
Write the ASCII text "hwlat" into the current_tracer file of the tracing system
(mounted at /sys/kernel/tracing or /sys/kernel/tracing). It is possible to
redefine the threshold in microseconds (us) above which latency spikes will
be taken into account.
Example::
# echo hwlat > /sys/kernel/tracing/current_tracer
# echo 100 > /sys/kernel/tracing/tracing_thresh
The /sys/kernel/tracing/hwlat_detector interface contains the following files:
- width - time period to sample with CPUs held (usecs)
must be less than the total window size (enforced)
- window - total period of sampling, width being inside (usecs)
By default the width is set to 500,000 and window to 1,000,000, meaning that
for every 1,000,000 usecs (1s) the hwlat detector will spin for 500,000 usecs
(0.5s). If tracing_thresh contains zero when hwlat tracer is enabled, it will
change to a default of 10 usecs. If any latencies that exceed the threshold is
observed then the data will be written to the tracing ring buffer.
The minimum sleep time between periods is 1 millisecond. Even if width
is less than 1 millisecond apart from window, to allow the system to not
be totally starved.
If tracing_thresh was zero when hwlat detector was started, it will be set
back to zero if another tracer is loaded. Note, the last value in
tracing_thresh that hwlat detector had will be saved and this value will
be restored in tracing_thresh if it is still zero when hwlat detector is
started again.Tracing 파일과 thread mode
69-88`hwlat_detector`가 사용하는 tracing directory 파일은 threshold와 관측 maximum, CPU mask, sampling 폭과 window, thread 배치 mode를 제어한다.
파일역할
`tracing_threshold`고려할 최소 latency(usecs)
`tracing_max_latency`실제로 관측한 최대 hardware latency(usecs)
`tracing_cpumask`hwlat thread가 이동할 CPU 집합
`hwlat_detector/width`window 안에서 spin할 시간(usecs)
`hwlat_detector/window`width 실행 사이의 전체 주기(usecs)
`hwlat_detector/mode`thread 배치 mode
측정 조건과 결과를 담당하는 파일을 구분한다.
기본적으로 hwlat detector kernel thread 하나가 새 window를 시작할 때마다 `tracing_cpumask`의 CPU를 round-robin으로 이동한다. `mode`를 바꾸면 강제 이동을 끄거나 CPU마다 thread를 하나씩 만들 수 있다.
Mode동작
`none`강제 migration을 하지 않음
`round-robin`cpumask의 각 CPU를 차례로 이동, 기본값
`per-cpu`tracing_cpumask의 CPU마다 thread 하나 생성
CPU mask를 순회하는 방식을 선택한다.
The following tracing directory files are used by the hwlat_detector:
in /sys/kernel/tracing:
- tracing_threshold - minimum latency value to be considered (usecs)
- tracing_max_latency - maximum hardware latency actually observed (usecs)
- tracing_cpumask - the CPUs to move the hwlat thread across
- hwlat_detector/width - specified amount of time to spin within window (usecs)
- hwlat_detector/window - amount of time between (width) runs (usecs)
- hwlat_detector/mode - the thread mode
By default, one hwlat detector's kernel thread will migrate across each CPU
specified in cpumask at the beginning of a new window, in a round-robin
fashion. This behavior can be changed by changing the thread mode,
the available options are:
- none: do not force migration
- round-robin: migrate across each CPU specified in cpumask [default]
- per-cpu: create one thread for each cpu in tracing_cpumask
요약·해설
hwlat_detector.rst:1-88Interrupt를 차단한 TSC polling으로 SMI와 firmware가 만드는 hardware latency를 검출하고 sampling 주기, threshold, CPU migration mode를 설정하는 방법을 설명합니다.