요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
=======================
Intel Powerclamp Driver
=======================
By:
- Arjan van de Ven <[email protected]>
- Jacob Pan <[email protected]>
.. Contents:
(*) Introduction
- Goals and Objectives
(*) Theory of Operation
- Idle Injection
- Calibration
(*) Performance Analysis
- Effectiveness and Limitations
- Power vs Performance
- Scalability
- Calibration
- Comparison with Alternative Techniques
(*) Usage and Interfaces
- Generic Thermal Layer (sysfs)
- Kernel APIs (TBD)
(*) Module Parameters
INTRODUCTION
============
Consider the situation where a system’s power consumption must be
reduced at runtime, due to power budget, thermal constraint, or noise
level, and where active cooling is not preferred. Software managed
passive power reduction must be performed to prevent the hardware
actions that are designed for catastrophic scenarios.
Currently, P-states, T-states (clock modulation), and CPU offlining
are used for CPU throttling.
On Intel CPUs, C-states provide effective power reduction, but so far
they’re only used opportunistically, based on workload. With the
development of intel_powerclamp driver, the method of synchronizing
idle injection across all online CPU threads was introduced. The goal
is to achieve forced and controllable C-state residency.
Test/Analysis has been made in the areas of power, performance,
scalability, and user experience. In many cases, clear advantage is
shown over taking the CPU offline or modulating the CPU clock.
THEORY OF OPERATION
===================
Idle Injection
--------------
On modern Intel processors (Nehalem or later), package level C-state
residency is available in MSRs, thus also available to the kernel.
These MSRs are::
#define MSR_PKG_C2_RESIDENCY 0x60D
#define MSR_PKG_C3_RESIDENCY 0x3F8
#define MSR_PKG_C6_RESIDENCY 0x3F9
#define MSR_PKG_C7_RESIDENCY 0x3FA
If the kernel can also inject idle time to the system, then a
closed-loop control system can be established that manages package
level C-state. The intel_powerclamp driver is conceived as such a
control system, where the target set point is a user-selected idle
ratio (based on power reduction), and the error is the difference
between the actual package level C-state residency ratio and the target idle
ratio.
Injection is controlled by high priority kernel threads, spawned for
each online CPU.
These kernel threads, with SCHED_FIFO class, are created to perform
clamping actions of controlled duty ratio and duration. Each per-CPU
thread synchronizes its idle time and duration, based on the rounding
of jiffies, so accumulated errors can be prevented to avoid a jittery
effect. Threads are also bound to the CPU such that they cannot be
migrated, unless the CPU is taken offline. In this case, threads
belong to the offlined CPUs will be terminated immediately.
Running as SCHED_FIFO and relatively high priority, also allows such
scheme to work for both preemptible and non-preemptible kernels.
Alignment of idle time around jiffies ensures scalability for HZ
values. This effect can be better visualized using a Perf timechart.
The following diagram shows the behavior of kernel thread
kidle_inject/cpu. During idle injection, it runs monitor/mwait idle
for a given "duration", then relinquishes the CPU to other tasks,
until the next time interval.
The NOHZ schedule tick is disabled during idle time, but interrupts
are not masked. Tests show that the extra wakeups from scheduler tick
have a dramatic impact on the effectiveness of the powerclamp driver
on large scale systems (Westmere system with 80 processors).
::
CPU0
____________ ____________
kidle_inject/0 | sleep | mwait | sleep |
_________| |________| |_______
duration
CPU1
____________ ____________
kidle_inject/1 | sleep | mwait | sleep |
_________| |________| |_______
^
|
|
roundup(jiffies, interval)
Only one CPU is allowed to collect statistics and update global
control parameters. This CPU is referred to as the controlling CPU in
this document. The controlling CPU is elected at runtime, with a
policy that favors BSP, taking into account the possibility of a CPU
hot-plug.
In terms of dynamics of the idle control system, package level idle
time is considered largely as a non-causal system where its behavior
cannot be based on the past or current input. Therefore, the
intel_powerclamp driver attempts to enforce the desired idle time
instantly as given input (target idle ratio). After injection,
powerclamp monitors the actual idle for a given time window and adjust
the next injection accordingly to avoid over/under correction.
When used in a causal control system, such as a temperature control,
it is up to the user of this driver to implement algorithms where
past samples and outputs are included in the feedback. For example, a
PID-based thermal controller can use the powerclamp driver to
maintain a desired target temperature, based on integral and
derivative gains of the past samples.
Calibration
-----------
During scalability testing, it is observed that synchronized actions
among CPUs become challenging as the number of cores grows. This is
also true for the ability of a system to enter package level C-states.
To make sure the intel_powerclamp driver scales well, online
calibration is implemented. The goals for doing such a calibration
are:
a) determine the effective range of idle injection ratio
b) determine the amount of compensation needed at each target ratio
Compensation to each target ratio consists of two parts:
a) steady state error compensation
This is to offset the error occurring when the system can
enter idle without extra wakeups (such as external interrupts).
b) dynamic error compensation
When an excessive amount of wakeups occurs during idle, an
additional idle ratio can be added to quiet interrupts, by
slowing down CPU activities.
A debugfs file is provided for the user to examine compensation
progress and results, such as on a Westmere system::
[jacob@nex01 ~]$ cat
/sys/kernel/debug/intel_powerclamp/powerclamp_calib
controlling cpu: 0
pct confidence steady dynamic (compensation)
0 0 0 0
1 1 0 0
2 1 1 0
3 3 1 0
4 3 1 0
5 3 1 0
6 3 1 0
7 3 1 0
8 3 1 0
...
30 3 2 0
31 3 2 0
32 3 1 0
33 3 2 0
34 3 1 0
35 3 2 0
36 3 1 0
37 3 2 0
38 3 1 0
39 3 2 0
40 3 3 0
41 3 1 0
42 3 2 0
43 3 1 0
44 3 1 0
45 3 2 0
46 3 3 0
47 3 0 0
48 3 2 0
49 3 3 0
Calibration occurs during runtime. No offline method is available.
Steady state compensation is used only when confidence levels of all
adjacent ratios have reached satisfactory level. A confidence level
is accumulated based on clean data collected at runtime. Data
collected during a period without extra interrupts is considered
clean.
To compensate for excessive amounts of wakeup during idle, additional
idle time is injected when such a condition is detected. Currently,
we have a simple algorithm to double the injection ratio. A possible
enhancement might be to throttle the offending IRQ, such as delaying
EOI for level triggered interrupts. But it is a challenge to be
non-intrusive to the scheduler or the IRQ core code.
CPU Online/Offline
------------------
Per-CPU kernel threads are started/stopped upon receiving
notifications of CPU hotplug activities. The intel_powerclamp driver
keeps track of clamping kernel threads, even after they are migrated
to other CPUs, after a CPU offline event.
Performance Analysis
====================
This section describes the general performance data collected on
multiple systems, including Westmere (80P) and Ivy Bridge (4P, 8P).
Effectiveness and Limitations
-----------------------------
The maximum range that idle injection is allowed is capped at 50
percent. As mentioned earlier, since interrupts are allowed during
forced idle time, excessive interrupts could result in less
effectiveness. The extreme case would be doing a ping -f to generated
flooded network interrupts without much CPU acknowledgement. In this
case, little can be done from the idle injection threads. In most
normal cases, such as scp a large file, applications can be throttled
by the powerclamp driver, since slowing down the CPU also slows down
network protocol processing, which in turn reduces interrupts.
When control parameters change at runtime by the controlling CPU, it
may take an additional period for the rest of the CPUs to catch up
with the changes. During this time, idle injection is out of sync,
thus not able to enter package C- states at the expected ratio. But
this effect is minor, in that in most cases change to the target
ratio is updated much less frequently than the idle injection
frequency.
Scalability
-----------
Tests also show a minor, but measurable, difference between the 4P/8P
Ivy Bridge system and the 80P Westmere server under 50% idle ratio.
More compensation is needed on Westmere for the same amount of
target idle ratio. The compensation also increases as the idle ratio
gets larger. The above reason constitutes the need for the
calibration code.
On the IVB 8P system, compared to an offline CPU, powerclamp can
achieve up to 40% better performance per watt. (measured by a spin
counter summed over per CPU counting threads spawned for all running
CPUs).
Usage and Interfaces
====================
The powerclamp driver is registered to the generic thermal layer as a
cooling device. Currently, it’s not bound to any thermal zones::
jacob@chromoly:/sys/class/thermal/cooling_device14$ grep . *
cur_state:0
max_state:50
type:intel_powerclamp
cur_state allows user to set the desired idle percentage. Writing 0 to
cur_state will stop idle injection. Writing a value between 1 and
max_state will start the idle injection. Reading cur_state returns the
actual and current idle percentage. This may not be the same value
set by the user in that current idle percentage depends on workload
and includes natural idle. When idle injection is disabled, reading
cur_state returns value -1 instead of 0 which is to avoid confusing
100% busy state with the disabled state.
Example usage:
- To inject 25% idle time::
$ sudo sh -c "echo 25 > /sys/class/thermal/cooling_device80/cur_state
If the system is not busy and has more than 25% idle time already,
then the powerclamp driver will not start idle injection. Using Top
will not show idle injection kernel threads.
If the system is busy (spin test below) and has less than 25% natural
idle time, powerclamp kernel threads will do idle injection. Forced
idle time is accounted as normal idle in that common code path is
taken as the idle task.
In this example, 24.1% idle is shown. This helps the system admin or
user determine the cause of slowdown, when a powerclamp driver is in action::
Tasks: 197 total, 1 running, 196 sleeping, 0 stopped, 0 zombie
Cpu(s): 71.2%us, 4.7%sy, 0.0%ni, 24.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3943228k total, 1689632k used, 2253596k free, 74960k buffers
Swap: 4087804k total, 0k used, 4087804k free, 945336k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3352 jacob 20 0 262m 644 428 S 286 0.0 0:17.16 spin
3341 root -51 0 0 0 0 D 25 0.0 0:01.62 kidle_inject/0
3344 root -51 0 0 0 0 D 25 0.0 0:01.60 kidle_inject/3
3342 root -51 0 0 0 0 D 25 0.0 0:01.61 kidle_inject/1
3343 root -51 0 0 0 0 D 25 0.0 0:01.60 kidle_inject/2
2935 jacob 20 0 696m 125m 35m S 5 3.3 0:31.11 firefox
1546 root 20 0 158m 20m 6640 S 3 0.5 0:26.97 Xorg
2100 jacob 20 0 1223m 88m 30m S 3 2.3 0:23.68 compiz
Tests have shown that by using the powerclamp driver as a cooling
device, a PID based userspace thermal controller can manage to
control CPU temperature effectively, when no other thermal influence
is added. For example, a UltraBook user can compile the kernel under
certain temperature (below most active trip points).
Module Parameters
=================
``cpumask`` (RW)
A bit mask of CPUs to inject idle. The format of the bitmask is same as
used in other subsystems like in /proc/irq/\*/smp_affinity. The mask is
comma separated 32 bit groups. Each CPU is one bit. For example for a 256
CPU system the full mask is:
ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
The rightmost mask is for CPU 0-32.
``max_idle`` (RW)
Maximum injected idle time to the total CPU time ratio in percent range
from 1 to 100. Even if the cooling device max_state is always 100 (100%),
this parameter allows to add a max idle percent limit. The default is 50,
to match the current implementation of powerclamp driver. Also doesn't
allow value more than 75, if the cpumask includes every CPU present in
the system.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
Intel Powerclamp Driver
1-30Arjan van de Ven과 Jacob Pan이 작성한 Intel Powerclamp driver 문서입니다. introduction, idle injection과 calibration 동작 원리, performance·scalability 분석, generic thermal layer interface, module parameter를 다룹니다.
INTRODUCTION
31-53power budget, thermal constraint, noise level 때문에 runtime power consumption을 줄여야 하고 active cooling을 선호하지 않는 상황을 다룹니다. catastrophic scenario용 hardware action이 시작되기 전에 software-managed passive power reduction을 수행해야 합니다.
CPU throttling에는 P-states, T-states(clock modulation), CPU offlining이 사용됩니다. Intel CPU의 C-state는 효과적으로 power를 줄이지만 이전에는 workload에 따라 opportunistic하게만 사용했습니다.
`intel_powerclamp` driver는 모든 online CPU thread의 idle injection을 동기화해 강제 가능하고 제어 가능한 C-state residency를 목표로 합니다. power, performance, scalability, user experience 분석에서 많은 경우 CPU offline이나 clock modulation보다 분명한 이점을 보였습니다.
THEORY OF OPERATION
54-56idle injection과 runtime calibration으로 closed-loop control을 구성합니다.
Idle Injection
57-141Nehalem 이후 Intel processor는 package-level C-state residency를 MSR로 제공하므로 kernel이 이를 읽을 수 있습니다.
#define MSR_PKG_C2_RESIDENCY 0x60D
#define MSR_PKG_C3_RESIDENCY 0x3F8
#define MSR_PKG_C6_RESIDENCY 0x3F9
#define MSR_PKG_C7_RESIDENCY 0x3FAkernel이 system에 idle time을 주입하면 package-level C-state를 관리하는 closed-loop control을 만들 수 있습니다. target set point는 사용자가 고른 idle ratio이고 error는 실제 package C-state residency ratio와 target idle ratio의 차이입니다.
각 online CPU에 생성된 high-priority kernel thread가 injection을 제어합니다. `SCHED_FIFO` class의 CPU별 thread는 제어된 duty ratio와 duration으로 clamping하고, 누적 오차와 jitter를 피하려고 jiffies 반올림 기준으로 idle 시작과 길이를 동기화합니다. CPU에 bind되어 offline되지 않는 한 migrate하지 않으며 offline 시 즉시 종료됩니다.
높은 priority의 `SCHED_FIFO`라 preemptible·non-preemptible kernel 모두에서 동작합니다. jiffies 주변 정렬은 여러 HZ 값에서 scalability를 확보합니다. `kidle_inject/cpu`는 주기마다 sleep 뒤 지정 `duration` 동안 `monitor/mwait` idle을 수행하고 다음 interval까지 CPU를 다른 task에 넘깁니다.
CPU0과 CPU1의 kidle_inject thread가 같은 jiffies 기반 interval 경계에 mwait 구간을 배치합니다.
idle 동안 `NOHZ` scheduler tick은 비활성화되지만 interrupt는 mask하지 않습니다. scheduler tick의 추가 wakeup은 80-processor Westmere 같은 대형 system에서 driver 효과를 크게 낮춥니다.
통계 수집과 global control parameter 갱신은 controlling CPU 하나만 담당합니다. runtime에 BSP를 선호하되 CPU hotplug 가능성을 고려해 선출합니다.
package idle time은 과거·현재 input으로 behavior를 정하기 어려운 non-causal system으로 간주합니다. driver는 target idle ratio를 즉시 강제하고 일정 window의 실제 idle을 관찰해 다음 injection을 보정합니다. temperature 같은 causal control에서는 사용자가 과거 sample과 output을 feedback에 포함해야 하며 PID thermal controller가 integral·derivative gain으로 target temperature를 유지할 수 있습니다.
Calibration
142-220core 수가 늘수록 CPU 사이 동기화와 package C-state 진입이 어려워집니다. online calibration은 idle injection ratio의 effective range와 target ratio별 compensation 양을 결정해 scalability를 확보합니다.
| compensation | 목적 |
|---|---|
| steady-state error | external interrupt 같은 추가 wakeup 없이 system이 idle에 들어갈 때 생기는 오차 상쇄 |
| dynamic error | idle 중 wakeup이 과도하면 CPU 활동을 늦추고 interrupt를 잠재우도록 idle ratio 추가 |
debugfs에서 calibration 진행과 결과를 확인할 수 있습니다.
[jacob@nex01 ~]$ cat
/sys/kernel/debug/intel_powerclamp/powerclamp_calib
controlling cpu: 0
pct confidence steady dynamic (compensation)
0 0 0 0
1 1 0 0
2 1 1 0
3 3 1 0
4 3 1 0
5 3 1 0
6 3 1 0
7 3 1 0
8 3 1 0
...
30 3 2 0
31 3 2 0
32 3 1 0
33 3 2 0
34 3 1 0
35 3 2 0
36 3 1 0
37 3 2 0
38 3 1 0
39 3 2 0
40 3 3 0
41 3 1 0
42 3 2 0
43 3 1 0
44 3 1 0
45 3 2 0
46 3 3 0
47 3 0 0
48 3 2 0
49 3 3 0calibration은 runtime에만 수행되고 offline method는 없습니다. 인접한 모든 ratio의 confidence가 충분할 때만 steady-state compensation을 사용합니다. extra interrupt가 없는 기간의 clean data를 누적해 confidence를 계산합니다.
idle 중 wakeup이 과도하면 추가 idle time을 주입합니다. 현재 단순 algorithm은 injection ratio를 두 배로 만듭니다. level-triggered interrupt의 EOI를 지연해 문제 IRQ를 throttle하는 개선안이 가능하지만 scheduler나 IRQ core에 비침투적으로 구현하기 어렵습니다.
CPU Online/Offline
221-228CPU hotplug notification을 받으면 CPU별 kernel thread를 시작하거나 중지합니다. CPU offline event 뒤 thread가 다른 CPU로 migrate된 경우에도 driver가 clamping thread를 추적합니다.
Performance Analysis
229-233Westmere 80P와 Ivy Bridge 4P·8P를 포함한 여러 system에서 수집한 일반 performance data를 설명합니다.
Effectiveness and Limitations
234-253허용하는 idle injection 최대 범위는 50%입니다. forced idle 동안 interrupt를 허용하므로 과도한 interrupt는 효과를 낮춥니다. `ping -f`로 CPU acknowledgement가 거의 없는 network interrupt flood를 만들면 idle injection thread가 할 수 있는 일이 적습니다.
큰 file을 `scp`하는 일반 상황에서는 CPU를 늦추면 network protocol processing과 interrupt도 줄어 application을 throttle할 수 있습니다.
controlling CPU가 runtime control parameter를 바꾸면 나머지 CPU가 따라잡는 데 한 period가 더 걸릴 수 있습니다. 그동안 injection이 out of sync라 기대 ratio로 package C-state에 들어가지 못하지만 target ratio 변경 빈도가 injection 빈도보다 훨씬 낮아 영향은 작습니다.
Scalability
254-26750% idle ratio에서 Ivy Bridge 4P·8P와 Westmere 80P 사이에 작지만 측정 가능한 차이가 있습니다. 같은 target ratio에 Westmere가 더 많은 compensation을 요구하고 ratio가 커질수록 compensation도 늘어 calibration code가 필요합니다.
IVB 8P system에서는 CPU offline 방식과 비교해 CPU별 counting thread의 spin counter 합으로 측정한 watt당 performance가 최대 40% 더 좋았습니다.
Usage and Interfaces
268-326driver는 generic thermal layer에 cooling device로 등록되며 현재 특정 thermal zone에 bind되지는 않습니다.
jacob@chromoly:/sys/class/thermal/cooling_device14$ grep . *
cur_state:0
max_state:50
type:intel_powerclamp`cur_state`는 원하는 idle percentage를 설정합니다. `0`을 쓰면 injection을 멈추고 `1`부터 `max_state` 사이 값은 시작합니다. 읽은 값은 workload와 natural idle을 포함한 실제 현재 idle percentage라 사용자가 쓴 값과 다를 수 있습니다. 비활성화 상태는 100% busy의 `0`과 혼동하지 않도록 `-1`을 반환합니다.
idle time 25%를 주입하는 예입니다.
$ sudo sh -c "echo 25 > /sys/class/thermal/cooling_device80/cur_statesystem이 이미 25%보다 많이 idle이면 driver는 injection을 시작하지 않아 Top에도 kernel thread가 보이지 않습니다. busy system에서 natural idle이 25%보다 적으면 thread가 injection하며 common idle-task code path를 사용하므로 forced idle도 normal idle로 계산됩니다.
다음 Top 예에서는 24.1% idle과 `kidle_inject` thread가 보입니다.
Tasks: 197 total, 1 running, 196 sleeping, 0 stopped, 0 zombie
Cpu(s): 71.2%us, 4.7%sy, 0.0%ni, 24.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3943228k total, 1689632k used, 2253596k free, 74960k buffers
Swap: 4087804k total, 0k used, 4087804k free, 945336k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3352 jacob 20 0 262m 644 428 S 286 0.0 0:17.16 spin
3341 root -51 0 0 0 0 D 25 0.0 0:01.62 kidle_inject/0
3344 root -51 0 0 0 0 D 25 0.0 0:01.60 kidle_inject/3
3342 root -51 0 0 0 0 D 25 0.0 0:01.61 kidle_inject/1
3343 root -51 0 0 0 0 D 25 0.0 0:01.60 kidle_inject/2
2935 jacob 20 0 696m 125m 35m S 5 3.3 0:31.11 firefox
1546 root 20 0 158m 20m 6640 S 3 0.5 0:26.97 Xorg
2100 jacob 20 0 1223m 88m 30m S 3 2.3 0:23.68 compiztest에서는 다른 thermal 영향이 없을 때 powerclamp cooling device와 PID-based userspace thermal controller로 CPU temperature를 효과적으로 제어했습니다. 예를 들어 UltraBook에서 active trip point보다 낮은 일정 temperature로 kernel compile을 수행할 수 있습니다.
Module Parameters
327-345`cpumask`(RW)는 idle을 주입할 CPU bit mask입니다. `/proc/irq/*/smp_affinity`와 같은 comma-separated 32-bit group format이며 CPU 하나가 bit 하나입니다. 256-CPU full mask는 `ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff`이고 가장 오른쪽 group이 CPU 0-31입니다.
`max_idle`(RW)은 total CPU time에 대한 최대 injected idle percentage이며 범위는 1~100입니다. cooling device `max_state`가 항상 100이어도 이 parameter로 상한을 둡니다. 기본값은 현재 구현에 맞춘 50이며, `cpumask`가 system의 모든 CPU를 포함하면 75보다 큰 값을 허용하지 않습니다.
요약과 해설
intel_powerclamp.rst:1-345intel_powerclamp는 CPU frequency만 낮추는 대신 모든 online CPU에 idle을 동기 주입해 package C-state residency를 직접 만듭니다. interrupt와 natural idle 때문에 target과 actual 값이 달라질 수 있어 runtime calibration과 feedback가 핵심입니다.