Scheduler statistics 필드 읽기 | Linux 6.18.37

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

통계 ABI와 version

sched-stats.rst:1-56

/proc/schedstat 형식은 version과 timestamp를 먼저 제공합니다. 필드가 추가되거나 의미가 바뀔 수 있으므로 parser는 열 위치를 고정하기 전에 version을 확인해야 합니다. 단위도 필드별로 jiffy, nanosecond 또는 count가 섞일 수 있습니다.

CPU와 sched_domain 통계

sched-stats.rst:57-189

schedstat을 해석하는 세 층

범위관찰 대상대표 질문

CPUrunqueue activity와 scheduling count이 CPU가 얼마나 바쁘고 얼마나 자주 전환했는가

Domainidle/busy/newidle balancing attempt와 결과어느 topology 단계에서 balancing이 실패하는가

Taskrun time, wait time, timeslice count특정 task가 CPU를 기다린 시간은 얼마인가

숫자 하나보다 CPU → domain → task 통계를 같은 구간에서 함께 비교해야 병목 위치를 찾을 수 있습니다.

domain 행은 topology 단계별 load balancing 시도, 성공, 이동 task 수와 imbalance를 기록합니다. CPU별 domain 수와 이름은 topology와 cpuset 구성에 따라 달라질 수 있으므로 CPU 번호만으로 행을 합치지 말고 domain mask를 함께 보아야 합니다.

/proc/PID/schedstat

sched-stats.rst:190-206

task schedstat은 CPU에서 실행한 시간, runqueue에서 기다린 시간, 실행된 timeslice 횟수를 제공합니다. 두 번 읽은 값의 차이를 interval로 나누면 관측 구간의 실행률과 평균 대기 시간을 계산할 수 있습니다. process 전체가 아니라 thread별 값이라는 점에 주의해야 합니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기

1 ====================

2 Scheduler Statistics

3 ====================

5 Version 17 of schedstats removed 'lb_imbalance' field as it has no

6 significance anymore and instead added more relevant fields namely

7 'lb_imbalance_load', 'lb_imbalance_util', 'lb_imbalance_task' and

8 'lb_imbalance_misfit'. The domain field prints the name of the

9 corresponding sched domain from this version onwards.

11 Version 16 of schedstats changed the order of definitions within

12 'enum cpu_idle_type', which changed the order of [CPU_MAX_IDLE_TYPES]

13 columns in show_schedstat(). In particular the position of CPU_IDLE

14 and __CPU_NOT_IDLE changed places. The size of the array is unchanged.

16 Version 15 of schedstats dropped counters for some sched_yield:

17 yld_exp_empty, yld_act_empty and yld_both_empty. Otherwise, it is

18 identical to version 14. Details are available at

	https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/scheduler/sched-stats.txt?id=1e1dbb259c79b

22 Version 14 of schedstats includes support for sched_domains, which hit the

23 mainline kernel in 2.6.20 although it is identical to the stats from version

24 12 which was in the kernel from 2.6.13-2.6.19 (version 13 never saw a kernel

25 release). Some counters make more sense to be per-runqueue; other to be

26 per-domain. Note that domains (and their associated information) will only

27 be pertinent and available on machines utilizing CONFIG_SMP.

29 In version 14 of schedstat, there is at least one level of domain

30 statistics for each cpu listed, and there may well be more than one

31 domain. Domains have no particular names in this implementation, but

32 the highest numbered one typically arbitrates balancing across all the

33 cpus on the machine, while domain0 is the most tightly focused domain,

34 sometimes balancing only between pairs of cpus. At this time, there

35 are no architectures which need more than three domain levels. The first

36 field in the domain stats is a bit map indicating which cpus are affected

37 by that domain. Details are available at

39 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/sched-stats.txt?id=b762f3ffb797c

41 The schedstat documentation is maintained version 10 onwards and is not

42 updated for version 11 and 12. The details for version 10 are available at

44 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/sched-stats.txt?id=1da177e4c3f4

46 These fields are counters, and only increment. Programs which make use

47 of these will need to start with a baseline observation and then calculate

48 the change in the counters at each subsequent observation. A perl script

49 which does this for many of the fields is available at

51 http://eaglet.pdxhosts.com/rick/linux/schedstat/

53 Note that any such script will necessarily be version-specific, as the main

54 reason to change versions is changes in the output format. For those wishing

55 to write their own scripts, the fields are described here.

57 CPU statistics

58 --------------

59 cpu<N> 1 2 3 4 5 6 7 8 9

61 First field is a sched_yield() statistic:

63 1) # of times sched_yield() was called

65 Next three are schedule() statistics:

67 2) This field is a legacy array expiration count field used in the O(1)

68 scheduler. We kept it for ABI compatibility, but it is always set to zero.

69 3) # of times schedule() was called

70 4) # of times schedule() left the processor idle

72 Next two are try_to_wake_up() statistics:

74 5) # of times try_to_wake_up() was called

75 6) # of times try_to_wake_up() was called to wake up the local cpu

77 Next three are statistics describing scheduling latency:

79 7) sum of all time spent running by tasks on this processor (in nanoseconds)

80 8) sum of all time spent waiting to run by tasks on this processor (in

81 nanoseconds)

82 9) # of timeslices run on this cpu

85 Domain statistics

86 -----------------

87 One of these is produced per domain for each cpu described. (Note that if

88 CONFIG_SMP is not defined, *no* domains are utilized and these lines

89 will not appear in the output.)

domain<N> <name> <cpumask> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

93 The <name> field prints the name of the sched domain and is only supported

94 with schedstat version >= 17. On previous versions, <cpumask> is the first

95 field.

97 The <cpumask> field is a bit mask indicating what cpus this domain operates

98 over.

100 The next 33 are a variety of sched_balance_rq() statistics in grouped into types

101 of idleness (busy, idle and newly idle):

102

103 1) # of times in this domain sched_balance_rq() was called when the

104 cpu was busy

105 2) # of times in this domain sched_balance_rq() checked but found the

106 load did not require balancing when busy

107 3) # of times in this domain sched_balance_rq() tried to move one or

108 more tasks and failed, when the cpu was busy

109 4) Total imbalance in load in this domain when the cpu was busy

110 5) Total imbalance in utilization in this domain when the cpu was busy

111 6) Total imbalance in number of tasks in this domain when the cpu was busy

112 7) Total imbalance due to misfit tasks in this domain when the cpu was

113 busy

114 8) # of times in this domain detach_task() was called when busy

115 9) # of times in this domain detach_task() was called even though the

116 target task was cache-hot when busy

117 10) # of times in this domain sched_balance_rq() was called but did not

118 find a busier queue while the cpu was busy

119 11) # of times in this domain a busier queue was found while the cpu

120 was busy but no busier group was found

121

122 12) # of times in this domain sched_balance_rq() was called when the

123 cpu was idle

124 13) # of times in this domain sched_balance_rq() checked but found

125 the load did not require balancing when the cpu was idle

126 14) # of times in this domain sched_balance_rq() tried to move one or

127 more tasks and failed, when the cpu was idle

128 15) Total imbalance in load in this domain when the cpu was idle

129 16) Total imbalance in utilization in this domain when the cpu was idle

130 17) Total imbalance in number of tasks in this domain when the cpu was idle

131 18) Total imbalance due to misfit tasks in this domain when the cpu was

132 idle

133 19) # of times in this domain detach_task() was called when the cpu

134 was idle

135 20) # of times in this domain detach_task() was called even though

136 the target task was cache-hot when idle

137 21) # of times in this domain sched_balance_rq() was called but did

138 not find a busier queue while the cpu was idle

139 22) # of times in this domain a busier queue was found while the

140 cpu was idle but no busier group was found

141

142 23) # of times in this domain sched_balance_rq() was called when the

143 cpu was just becoming idle

144 24) # of times in this domain sched_balance_rq() checked but found the

145 load did not require balancing when the cpu was just becoming idle

146 25) # of times in this domain sched_balance_rq() tried to move one or more

147 tasks and failed, when the cpu was just becoming idle

148 26) Total imbalance in load in this domain when the cpu was just becoming

149 idle

150 27) Total imbalance in utilization in this domain when the cpu was just

151 becoming idle

152 28) Total imbalance in number of tasks in this domain when the cpu was just

153 becoming idle

154 29) Total imbalance due to misfit tasks in this domain when the cpu was

155 just becoming idle

156 30) # of times in this domain detach_task() was called when newly idle

157 31) # of times in this domain detach_task() was called even though the

158 target task was cache-hot when just becoming idle

159 32) # of times in this domain sched_balance_rq() was called but did not

160 find a busier queue while the cpu was just becoming idle

161 33) # of times in this domain a busier queue was found while the cpu

162 was just becoming idle but no busier group was found

163

164 Next three are active_load_balance() statistics:

165

166 34) # of times active_load_balance() was called

167 35) # of times active_load_balance() tried to move a task and failed

168 36) # of times active_load_balance() successfully moved a task

169

170 Next three are sched_balance_exec() statistics:

171

172 37) sbe_cnt is not used

173 38) sbe_balanced is not used

174 39) sbe_pushed is not used

175

176 Next three are sched_balance_fork() statistics:

177

178 40) sbf_cnt is not used

179 41) sbf_balanced is not used

180 42) sbf_pushed is not used

181

182 Next three are try_to_wake_up() statistics:

183

184 43) # of times in this domain try_to_wake_up() awoke a task that

185 last ran on a different cpu in this domain

186 44) # of times in this domain try_to_wake_up() moved a task to the

187 waking cpu because it was cache-cold on its own cpu anyway

188 45) # of times in this domain try_to_wake_up() started passive balancing

189

190 /proc/<pid>/schedstat

191 ---------------------

192 schedstats also adds a new /proc/<pid>/schedstat file to include some of

193 the same information on a per-process level. There are three fields in

194 this file correlating for that process to:

195

196 1) time spent on the cpu (in nanoseconds)

197 2) time spent waiting on a runqueue (in nanoseconds)

198 3) # of timeslices run on this cpu

199

200 A program could be easily written to make use of these extra fields to

201 report on how well a particular process or set of processes is faring

202 under the scheduler's policies. A simple version of such a program is

203 available at

204

205 http://eaglet.pdxhosts.com/rick/linux/schedstat/v12/latency.c

206

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

schedstats format version

1-55

schedstats version 17은 의미가 사라진 단일 lb_imbalance field를 제거하고 lb_imbalance_load, lb_imbalance_util, lb_imbalance_task, lb_imbalance_misfit을 추가했다. 이 version부터 domain field에 해당 sched domain 이름도 출력한다.

version 16은 enum cpu_idle_type 정의 순서를 바꾸어 show_schedstat()의 CPU_MAX_IDLE_TYPES column 순서가 달라졌다. 특히 CPU_IDLE과 __CPU_NOT_IDLE 위치가 서로 바뀌었으며 array 크기는 같다.

version 15는 sched_yield 관련 yld_exp_empty, yld_act_empty, yld_both_empty counter를 제거했고 나머지는 version 14와 같다. version 14는 kernel 2.6.20에 들어간 sched_domain 지원을 포함하며, version 12와 같은 통계 구조다. version 13은 정식 kernel release에 포함되지 않았다.

CONFIG_SMP system에서 CPU마다 하나 이상의 domain statistics가 출력된다. domain0은 보통 가장 좁은 CPU group을 balance하고 높은 번호 domain은 machine 전체에 가까운 범위를 다룬다. version 17 이전에는 이름이 없었으며 첫 field인 cpumask로 범위를 해석했다.

모든 field는 누적 counter이며 감소하지 않는다. 측정 program은 첫 snapshot을 baseline으로 저장하고 다음 snapshot과의 차이를 계산해야 한다. 출력 format 변경이 version 증가의 주된 이유이므로 parser는 schedstats version별 field layout을 사용해야 한다.

CPU statistics 9개 field

57-83

cpu<N> 1 2 3 4 5 6 7 8 9

번호	분류	의미
1	sched_yield	sched_yield() 호출 횟수
2	schedule	O(1) scheduler의 legacy array expiration count; ABI 호환을 위해 남아 있으며 항상 0
3	schedule	schedule() 호출 횟수
4	schedule	schedule() 결과 processor가 idle이 된 횟수
5	try_to_wake_up	try_to_wake_up() 호출 횟수
6	try_to_wake_up	local CPU를 깨우기 위한 try_to_wake_up() 호출 횟수
7	latency	이 CPU에서 task가 실제 실행한 총 시간(ns)
8	latency	이 CPU runqueue에서 task가 실행을 기다린 총 시간(ns)
9	latency	이 CPU에서 실행된 timeslice 수

field 7과 8의 delta를 같은 sampling interval에서 비교하면 CPU service time과 runqueue wait time 변화를 볼 수 있다. field 9 delta로 그 기간의 scheduling slice 수를 함께 해석한다.

domain statistics 기본 형식

85-99

각 CPU의 각 sched domain마다 한 줄이 출력된다. CONFIG_SMP가 없으면 sched domain을 사용하지 않아 domain 줄도 나타나지 않는다.

domain<N> <name> <cpumask> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

<name>은 schedstats version 17 이상에서 sched domain 이름을 출력한다. 이전 version에서는 <cpumask>가 첫 field다. cpumask는 이 domain이 balance 대상으로 삼는 CPU 집합의 bit mask다.

sched_balance_rq field 1-33

100-163

field 1-33은 CPU idle type별 sched_balance_rq() 통계다. busy, idle, newly idle 세 상태마다 같은 11개 의미가 반복된다.

상대 위치	busy 번호	idle 번호	newly-idle 번호	의미
1	1	12	23	해당 상태에서 sched_balance_rq() 호출 횟수
2	2	13	24	검사했으나 load상 balancing이 필요 없었던 횟수
3	3	14	25	하나 이상 task move를 시도했으나 실패한 횟수
4	4	15	26	load 기준 total imbalance
5	5	16	27	utilization 기준 total imbalance
6	6	17	28	task 수 기준 total imbalance
7	7	18	29	misfit task 때문에 생긴 total imbalance
8	8	19	30	detach_task() 호출 횟수
9	9	20	31	target task가 cache-hot인데도 detach_task()를 호출한 횟수
10	10	21	32	balance를 호출했지만 더 busy한 runqueue를 찾지 못한 횟수
11	11	22	33	더 busy한 runqueue는 찾았지만 더 busy한 group은 찾지 못한 횟수

busy는 CPU가 이미 task를 실행하는 상태, idle은 idle balancing, newly idle은 CPU가 막 idle로 전환되는 지점의 balancing이다. version 16 이전·이후를 비교할 때 enum cpu_idle_type 순서 변경을 반영해야 한다.

domain balance counter 묶음

busyidlenewly idle

01 field 1-11field 12-22field 23-33

02 실행 중 periodic balanceidle CPU balanceidle 진입 시 pull 시도

03 load/util/task/misfit imbalance같은 4 imbalance같은 4 imbalance

04 detach/cache-hot/no-busier-rq/group동일 counter동일 counter

같은 11개 counter가 CPU 상태에 따라 서로 다른 field 범위에 배치된다.

domain field 34-45와 per-process schedstat

164-205

번호	분류	의미
34	active_load_balance	active_load_balance() 호출 횟수
35	active_load_balance	task move 시도 실패 횟수
36	active_load_balance	task move 성공 횟수
37	sched_balance_exec	sbe_cnt; 현재 사용하지 않음
38	sched_balance_exec	sbe_balanced; 현재 사용하지 않음
39	sched_balance_exec	sbe_pushed; 현재 사용하지 않음
40	sched_balance_fork	sbf_cnt; 현재 사용하지 않음
41	sched_balance_fork	sbf_balanced; 현재 사용하지 않음
42	sched_balance_fork	sbf_pushed; 현재 사용하지 않음
43	try_to_wake_up	마지막 실행 CPU가 이 domain의 다른 CPU인 task를 깨운 횟수
44	try_to_wake_up	원래 CPU에서 cache-cold인 task를 waking CPU로 옮긴 횟수
45	try_to_wake_up	passive balancing을 시작한 횟수

schedstats는 process별 /proc/<pid>/schedstat도 제공한다. field는 해당 process의 CPU 실행 시간(ns), runqueue 대기 시간(ns), 실행된 timeslice 수 세 개다.

/proc/<pid>/schedstat

1) CPU에서 실행한 시간(ns)
2) runqueue에서 기다린 시간(ns)
3) 실행된 timeslice 수

sampling interval마다 세 field의 delta를 구하면 특정 process 또는 process 집합이 scheduler policy 아래에서 받은 CPU service와 대기 latency를 관찰할 수 있다. counter wrap과 process lifetime, format version을 함께 처리해야 한다.