← Documents Documentation/networking/pktgen.rst GitHub 원문 ↗

Linux 6.18.37 · Networking

Linux packet generator 사용법

CPU별 pktgen thread, procfs command, NIC tuning, packet parameter와 고급 protocol option 사용법입니다.

Source pathDocumentation/networking/pktgen.rst
Source versionLinux v6.18.37
TranslationDUJINLABS 전문 번역 + 해설

요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.

1. 요약·해설

원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.

요약·해설

pktgen.rst:1-422

Pktgen은 CPU별 kernel thread가 procfs 설정에 따라 고속 packet을 생성하는 network benchmark 도구입니다. NIC ring과 interrupt affinity를 함께 조정하고, device별 packet·주소·queue·protocol parameter를 설정하며, CPU locality와 일반 traffic에 미치는 영향을 구분해 측정해야 합니다.

Pktgen benchmark workflow
modprobe pktgenkpktgend_X에 device@X 추가Device pgset 구성NIC·IRQ affinity 조정pg_ctrl startParams / Current / Result 확인

Module load부터 결과 확인까지의 기본 절차입니다.

2. 영어 원문 전체

번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.

원문 전체 펼치기
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ====================================
4 HOWTO for the linux packet generator
5 ====================================
6
7 Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
8 or as a module. A module is preferred; modprobe pktgen if needed. Once
9 running, pktgen creates a thread for each CPU with affinity to that CPU.
10 Monitoring and controlling is done via /proc. It is easiest to select a
11 suitable sample script and configure that.
12
13 On a dual CPU::
14
15 ps aux | grep pkt
16 root 129 0.3 0.0 0 0 ? SW 2003 523:20 [kpktgend_0]
17 root 130 0.3 0.0 0 0 ? SW 2003 509:50 [kpktgend_1]
18
19
20 For monitoring and control pktgen creates::
21
22 /proc/net/pktgen/pgctrl
23 /proc/net/pktgen/kpktgend_X
24 /proc/net/pktgen/ethX
25
26
27 Tuning NIC for max performance
28 ==============================
29
30 The default NIC settings are (likely) not tuned for pktgen's artificial
31 overload type of benchmarking, as this could hurt the normal use-case.
32
33 Specifically increasing the TX ring buffer in the NIC::
34
35 # ethtool -G ethX tx 1024
36
37 A larger TX ring can improve pktgen's performance, while it can hurt
38 in the general case, 1) because the TX ring buffer might get larger
39 than the CPU's L1/L2 cache, 2) because it allows more queueing in the
40 NIC HW layer (which is bad for bufferbloat).
41
42 One should hesitate to conclude that packets/descriptors in the HW
43 TX ring cause delay. Drivers usually delay cleaning up the
44 ring-buffers for various performance reasons, and packets stalling
45 the TX ring might just be waiting for cleanup.
46
47 This cleanup issue is specifically the case for the driver ixgbe
48 (Intel 82599 chip). This driver (ixgbe) combines TX+RX ring cleanups,
49 and the cleanup interval is affected by the ethtool --coalesce setting
50 of parameter "rx-usecs".
51
52 For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6)::
53
54 # ethtool -C ethX rx-usecs 30
55
56
57 Kernel threads
58 ==============
59 Pktgen creates a thread for each CPU with affinity to that CPU.
60 Which is controlled through procfile /proc/net/pktgen/kpktgend_X.
61
62 Example: /proc/net/pktgen/kpktgend_0::
63
64 Running:
65 Stopped: eth4@0
66 Result: OK: add_device=eth4@0
67
68 Most important are the devices assigned to the thread.
69
70 The two basic thread commands are:
71
72 * add_device DEVICE@NAME -- adds a single device
73 * rem_device_all -- remove all associated devices
74
75 When adding a device to a thread, a corresponding procfile is created
76 which is used for configuring this device. Thus, device names need to
77 be unique.
78
79 To support adding the same device to multiple threads, which is useful
80 with multi queue NICs, the device naming scheme is extended with "@":
81 device@something
82
83 The part after "@" can be anything, but it is custom to use the thread
84 number.
85
86 Viewing devices
87 ===============
88
89 The Params section holds configured information. The Current section
90 holds running statistics. The Result is printed after a run or after
91 interruption. Example::
92
93 /proc/net/pktgen/eth4@0
94
95 Params: count 100000 min_pkt_size: 60 max_pkt_size: 60
96 frags: 0 delay: 0 clone_skb: 64 ifname: eth4@0
97 flows: 0 flowlen: 0
98 queue_map_min: 0 queue_map_max: 0
99 dst_min: 192.168.81.2 dst_max:
100 src_min: src_max:
101 src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
102 udp_src_min: 9 udp_src_max: 109 udp_dst_min: 9 udp_dst_max: 9
103 src_mac_count: 0 dst_mac_count: 0
104 Flags: UDPSRC_RND NO_TIMESTAMP QUEUE_MAP_CPU
105 Current:
106 pkts-sofar: 100000 errors: 0
107 started: 623913381008us stopped: 623913396439us idle: 25us
108 seq_num: 100001 cur_dst_mac_offset: 0 cur_src_mac_offset: 0
109 cur_saddr: 192.168.8.3 cur_daddr: 192.168.81.2
110 cur_udp_dst: 9 cur_udp_src: 42
111 cur_queue_map: 0
112 flows: 0
113 Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
114 6480562pps 3110Mb/sec (3110669760bps) errors: 0
115
116
117 Configuring devices
118 ===================
119 This is done via the /proc interface, and most easily done via pgset
120 as defined in the sample scripts.
121 You need to specify PGDEV environment variable to use functions from sample
122 scripts, i.e.::
123
124 export PGDEV=/proc/net/pktgen/eth4@0
125 source samples/pktgen/functions.sh
126
127 Examples::
128
129 pg_ctrl start starts injection.
130 pg_ctrl stop aborts injection. Also, ^C aborts generator.
131
132 pgset "clone_skb 1" sets the number of copies of the same packet
133 pgset "clone_skb 0" use single SKB for all transmits
134 pgset "burst 8" uses xmit_more API to queue 8 copies of the same
135 packet and update HW tx queue tail pointer once.
136 "burst 1" is the default
137 pgset "pkt_size 9014" sets packet size to 9014
138 pgset "frags 5" packet will consist of 5 fragments
139 pgset "count 200000" sets number of packets to send, set to zero
140 for continuous sends until explicitly stopped.
141
142 pgset "delay 5000" adds delay to hard_start_xmit(). nanoseconds
143
144 pgset "dst 10.0.0.1" sets IP destination address
145 (BEWARE! This generator is very aggressive!)
146
147 pgset "dst_min 10.0.0.1" Same as dst
148 pgset "dst_max 10.0.0.254" Set the maximum destination IP.
149 pgset "src_min 10.0.0.1" Set the minimum (or only) source IP.
150 pgset "src_max 10.0.0.254" Set the maximum source IP.
151 pgset "dst6 fec0::1" IPV6 destination address
152 pgset "src6 fec0::2" IPV6 source address
153 pgset "dstmac 00:00:00:00:00:00" sets MAC destination address
154 pgset "srcmac 00:00:00:00:00:00" sets MAC source address
155
156 pgset "queue_map_min 0" Sets the min value of tx queue interval
157 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
158 To select queue 1 of a given device,
159 use queue_map_min=1 and queue_map_max=1
160
161 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
162 The 'minimum' MAC is what you set with srcmac.
163
164 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
165 The 'minimum' MAC is what you set with dstmac.
166
167 pgset "flag [name]" Set a flag to determine behaviour. Current flags
168 are: IPSRC_RND # IP source is random (between min/max)
169 IPDST_RND # IP destination is random
170 UDPSRC_RND, UDPDST_RND,
171 MACSRC_RND, MACDST_RND
172 TXSIZE_RND, IPV6,
173 MPLS_RND, VID_RND, SVID_RND
174 FLOW_SEQ,
175 QUEUE_MAP_RND # queue map random
176 QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
177 UDPCSUM,
178 IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
179 NODE_ALLOC # node specific memory allocation
180 NO_TIMESTAMP # disable timestamping
181 SHARED # enable shared SKB
182 pgset 'flag ![name]' Clear a flag to determine behaviour.
183 Note that you might need to use single quote in
184 interactive mode, so that your shell wouldn't expand
185 the specified flag as a history command.
186
187 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet.
188
189 pgset "udp_src_min 9" set UDP source port min, If < udp_src_max, then
190 cycle through the port range.
191
192 pgset "udp_src_max 9" set UDP source port max.
193 pgset "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then
194 cycle through the port range.
195 pgset "udp_dst_max 9" set UDP destination port max.
196
197 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
198 outer label=16,middle label=32,
199 inner label=0 (IPv4 NULL)) Note that
200 there must be no spaces between the
201 arguments. Leading zeros are required.
202 Do not set the bottom of stack bit,
203 that's done automatically. If you do
204 set the bottom of stack bit, that
205 indicates that you want to randomly
206 generate that address and the flag
207 MPLS_RND will be turned on. You
208 can have any mix of random and fixed
209 labels in the label stack.
210
211 pgset "mpls 0" turn off mpls (or any invalid argument works too!)
212
213 pgset "vlan_id 77" set VLAN ID 0-4095
214 pgset "vlan_p 3" set priority bit 0-7 (default 0)
215 pgset "vlan_cfi 0" set canonical format identifier 0-1 (default 0)
216
217 pgset "svlan_id 22" set SVLAN ID 0-4095
218 pgset "svlan_p 3" set priority bit 0-7 (default 0)
219 pgset "svlan_cfi 0" set canonical format identifier 0-1 (default 0)
220
221 pgset "vlan_id 9999" > 4095 remove vlan and svlan tags
222 pgset "svlan 9999" > 4095 remove svlan tag
223
224
225 pgset "tos XX" set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
226 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
227
228 pgset "rate 300M" set rate to 300 Mb/s
229 pgset "ratep 1000000" set rate to 1Mpps
230
231 pgset "xmit_mode netif_receive" RX inject into stack netif_receive_skb()
232 Works with "burst" but not with "clone_skb".
233 Default xmit_mode is "start_xmit".
234
235 Sample scripts
236 ==============
237
238 A collection of tutorial scripts and helpers for pktgen is in the
239 samples/pktgen directory. The helper parameters.sh file support easy
240 and consistent parameter parsing across the sample scripts.
241
242 Usage example and help::
243
244 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2
245
246 Usage:::
247
248 ./pktgen_sample01_simple.sh [-vx] -i ethX
249
250 -i : ($DEV) output interface/device (required)
251 -s : ($PKT_SIZE) packet size
252 -d : ($DEST_IP) destination IP. CIDR (e.g. 198.18.0.0/15) is also allowed
253 -m : ($DST_MAC) destination MAC-addr
254 -p : ($DST_PORT) destination PORT range (e.g. 433-444) is also allowed
255 -t : ($THREADS) threads to start
256 -f : ($F_THREAD) index of first thread (zero indexed CPU number)
257 -c : ($SKB_CLONE) SKB clones send before alloc new SKB
258 -n : ($COUNT) num messages to send per thread, 0 means indefinitely
259 -b : ($BURST) HW level bursting of SKBs
260 -v : ($VERBOSE) verbose
261 -x : ($DEBUG) debug
262 -6 : ($IP6) IPv6
263 -w : ($DELAY) Tx Delay value (ns)
264 -a : ($APPEND) Script will not reset generator's state, but will append its config
265
266 The global variables being set are also listed. E.g. the required
267 interface/device parameter "-i" sets variable $DEV. Copy the
268 pktgen_sampleXX scripts and modify them to fit your own needs.
269
270
271 Interrupt affinity
272 ===================
273 Note that when adding devices to a specific CPU it is a good idea to
274 also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
275 to the same CPU. This reduces cache bouncing when freeing skbs.
276
277 Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
278 to the running threads CPU (directly from smp_processor_id()).
279
280 Enable IPsec
281 ============
282 Default IPsec transformation with ESP encapsulation plus transport mode
283 can be enabled by simply setting::
284
285 pgset "flag IPSEC"
286 pgset "flows 1"
287
288 To avoid breaking existing testbed scripts for using AH type and tunnel mode,
289 you can use "pgset spi SPI_VALUE" to specify which transformation mode
290 to employ.
291
292 Disable shared SKB
293 ==================
294 By default, SKBs sent by pktgen are shared (user count > 1).
295 To test with non-shared SKBs, remove the "SHARED" flag by simply setting::
296
297 pg_set "flag !SHARED"
298
299 However, if the "clone_skb" or "burst" parameters are configured, the skb
300 still needs to be held by pktgen for further access. Hence the skb must be
301 shared.
302
303 Current commands and configuration options
304 ==========================================
305
306 **Pgcontrol commands**::
307
308 start
309 stop
310 reset
311
312 **Thread commands**::
313
314 add_device
315 rem_device_all
316
317
318 **Device commands**::
319
320 count
321 clone_skb
322 burst
323 debug
324
325 frags
326 delay
327
328 src_mac_count
329 dst_mac_count
330
331 pkt_size
332 min_pkt_size
333 max_pkt_size
334
335 queue_map_min
336 queue_map_max
337 skb_priority
338
339 tos (ipv4)
340 traffic_class (ipv6)
341
342 mpls
343
344 udp_src_min
345 udp_src_max
346
347 udp_dst_min
348 udp_dst_max
349
350 node
351
352 flag
353 IPSRC_RND
354 IPDST_RND
355 UDPSRC_RND
356 UDPDST_RND
357 MACSRC_RND
358 MACDST_RND
359 TXSIZE_RND
360 IPV6
361 MPLS_RND
362 VID_RND
363 SVID_RND
364 FLOW_SEQ
365 QUEUE_MAP_RND
366 QUEUE_MAP_CPU
367 UDPCSUM
368 IPSEC
369 NODE_ALLOC
370 NO_TIMESTAMP
371 SHARED
372
373 spi (ipsec)
374
375 dst_min
376 dst_max
377
378 src_min
379 src_max
380
381 dst_mac
382 src_mac
383
384 clear_counters
385
386 src6
387 dst6
388 dst6_max
389 dst6_min
390
391 flows
392 flowlen
393
394 rate
395 ratep
396
397 xmit_mode <start_xmit|netif_receive>
398
399 vlan_cfi
400 vlan_id
401 vlan_p
402
403 svlan_cfi
404 svlan_id
405 svlan_p
406
407
408 References:
409
410 - ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
411 - ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
412
413 Paper from Linux-Kongress in Erlangen 2004.
414 - ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
415
416 Thanks to:
417
418 Grant Grundler for testing on IA-64 and parisc, Harald Welte, Lennert Buytenhek
419 Stephen Hemminger, Andi Kleen, Dave Miller and many others.
420
421
422 Good luck with the linux net-development.
423

3. 한국어 전문 번역

영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.

활성화와 procfs 제어 파일

1-26

`CONFIG_NET_PKTGEN`을 활성화하면 pktgen을 kernel 내장 또는 module로 build할 수 있습니다. Module 방식이 권장되며 필요하면 `modprobe pktgen`으로 load합니다. 실행되면 각 CPU마다 해당 CPU에 affinity가 고정된 thread를 하나씩 만듭니다.

Monitoring과 control은 `/proc`를 통해 수행합니다. 가장 쉬운 시작 방법은 목적에 맞는 sample script를 골라 parameter를 조정하는 것입니다. Dual-CPU 예에서는 `kpktgend_0`과 `kpktgend_1` kernel thread가 각각 보입니다.

전체 generator 제어는 `/proc/net/pktgen/pgctrl`, CPU별 thread 제어는 `/proc/net/pktgen/kpktgend_X`, device별 설정과 통계는 `/proc/net/pktgen/ethX`에서 수행합니다.

pktgen procfs interface
경로역할
/proc/net/pktgen/pgctrl전체 generator 시작·중지·reset
/proc/net/pktgen/kpktgend_XCPU X의 pktgen thread와 device 연결
/proc/net/pktgen/ethX 또는 device@name개별 device parameter와 실행 통계

제어 범위에 따라 파일이 나뉩니다.

pktgen 실행 구조
CONFIG_NET_PKTGEN / modprobeCPU별 kpktgend_XThread에 device 추가Device parameter 설정pgctrl start

CPU별 thread가 자신에게 배정된 device를 통해 packet을 생성합니다.

.. SPDX-License-Identifier: GPL-2.0

====================================
HOWTO for the linux packet generator
====================================

Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
or as a module.  A module is preferred; modprobe pktgen if needed.  Once
running, pktgen creates a thread for each CPU with affinity to that CPU.
Monitoring and controlling is done via /proc.  It is easiest to select a
suitable sample script and configure that.

On a dual CPU::

    ps aux | grep pkt
    root       129  0.3  0.0     0    0 ?        SW    2003 523:20 [kpktgend_0]
    root       130  0.3  0.0     0    0 ?        SW    2003 509:50 [kpktgend_1]


For monitoring and control pktgen creates::

        /proc/net/pktgen/pgctrl
        /proc/net/pktgen/kpktgend_X
        /proc/net/pktgen/ethX

최대 성능을 위한 NIC tuning

27-56

NIC의 기본 설정은 정상 workload를 위한 것이므로 pktgen의 인위적인 과부하 benchmark에 최적화되어 있지 않을 가능성이 큽니다. Benchmark 설정은 일반 사용 성능을 해칠 수 있습니다.

`ethtool -G ethX tx 1024`처럼 NIC의 TX ring을 늘리면 pktgen 성능이 좋아질 수 있습니다. 하지만 ring이 CPU L1/L2 cache보다 커질 수 있고 NIC hardware layer에 더 많은 queueing을 허용하여 bufferbloat를 악화하므로 일반 환경에는 불리할 수 있습니다.

Hardware TX ring 안의 packet이나 descriptor가 곧바로 지연 원인이라고 단정해서는 안 됩니다. Driver는 성능상 이유로 ring buffer cleanup을 늦추는 경우가 많으므로, ring에 머무는 packet은 단지 cleanup을 기다리는 것일 수 있습니다.

Intel 82599용 `ixgbe`는 TX와 RX ring cleanup을 결합하며 cleanup 간격은 ethtool coalesce의 `rx-usecs` 설정에 영향을 받습니다. 예를 들어 `ethtool -C ethX rx-usecs 30`은 대략 초당 33,000회 interrupt를 만듭니다.

pktgen용 NIC tuning의 상충
조정Pktgen 효과일반 환경 위험
TX ring 1024더 많은 descriptor로 높은 전송률Cache 초과와 bufferbloat
ixgbe rx-usecs 30TX+RX cleanup 빈도 조정약 33K interrupts/sec 부하

Benchmark throughput과 일반 latency 사이의 tradeoff입니다.

Tuning NIC for max performance
==============================

The default NIC settings are (likely) not tuned for pktgen's artificial
overload type of benchmarking, as this could hurt the normal use-case.

Specifically increasing the TX ring buffer in the NIC::

 # ethtool -G ethX tx 1024

A larger TX ring can improve pktgen's performance, while it can hurt
in the general case, 1) because the TX ring buffer might get larger
than the CPU's L1/L2 cache, 2) because it allows more queueing in the
NIC HW layer (which is bad for bufferbloat).

One should hesitate to conclude that packets/descriptors in the HW
TX ring cause delay.  Drivers usually delay cleaning up the
ring-buffers for various performance reasons, and packets stalling
the TX ring might just be waiting for cleanup.

This cleanup issue is specifically the case for the driver ixgbe
(Intel 82599 chip).  This driver (ixgbe) combines TX+RX ring cleanups,
and the cleanup interval is affected by the ethtool --coalesce setting
of parameter "rx-usecs".

For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6)::

 # ethtool -C ethX rx-usecs 30

CPU thread와 device 별칭

57-85

Pktgen은 CPU마다 그 CPU에 affinity가 고정된 thread를 만들고 `/proc/net/pktgen/kpktgend_X`로 제어합니다. Thread 파일은 현재 실행 중인 device, 중지된 device, 마지막 명령 결과를 보여 줍니다.

핵심 thread 명령은 `add_device DEVICE@NAME`과 `rem_device_all`입니다. 첫 명령은 device 하나를 thread에 추가하고, 둘째 명령은 연결된 모든 device를 제거합니다.

Thread에 device를 추가하면 해당 device를 설정할 procfile이 생성되므로 이름은 고유해야 합니다. Multi-queue NIC에서 같은 실제 device를 여러 thread에 추가할 수 있도록 `device@something` 형식을 사용합니다. `@` 뒤 문자열은 무엇이든 가능하지만 관례적으로 thread 번호를 씁니다.

Multi-queue device 배정
kpktgend_0add_device eth4@0/proc/net/pktgen/eth4@0
kpktgend_1add_device eth4@1/proc/net/pktgen/eth4@1

같은 NIC를 고유 별칭으로 여러 CPU thread에 연결합니다.

Kernel threads
==============
Pktgen creates a thread for each CPU with affinity to that CPU.
Which is controlled through procfile /proc/net/pktgen/kpktgend_X.

Example: /proc/net/pktgen/kpktgend_0::

 Running:
 Stopped: eth4@0
 Result: OK: add_device=eth4@0

Most important are the devices assigned to the thread.

The two basic thread commands are:

 * add_device DEVICE@NAME -- adds a single device
 * rem_device_all         -- remove all associated devices

When adding a device to a thread, a corresponding procfile is created
which is used for configuring this device. Thus, device names need to
be unique.

To support adding the same device to multiple threads, which is useful
with multi queue NICs, the device naming scheme is extended with "@":
device@something

The part after "@" can be anything, but it is custom to use the thread
number.

Device 상태와 실행 결과 읽기

86-116

Device procfile의 `Params` 절은 구성된 parameter를, `Current` 절은 실행 중 통계를 보관합니다. `Result`는 실행이 끝나거나 중단된 뒤 출력됩니다.

예제의 `Params`에는 packet count·size·fragment·delay·SKB clone 수, flow, TX queue 범위, source/destination IP와 MAC, UDP port 범위, 활성 flag가 기록됩니다. `Current`에는 지금까지의 packet·error 수, start/stop/idle 시간, sequence, 현재 주소와 port, queue map, flow 상태가 표시됩니다.

`Result`는 총 소요 시간과 clone·delay 내역, packet 수와 크기, packet per second, Mbit/s 및 bit/s, error 수를 요약합니다. 예제는 100,000개의 60-byte packet을 error 없이 약 6.48Mpps와 3110Mb/s로 전송한 결과입니다.

Device procfile 절
내용
Params입력한 packet, address, queue, flag 구성
Currentpkts-sofar, errors, 시간, 현재 주소·port·queue
Result완료 시간, pps, throughput, error 요약

설정·실시간 상태·최종 결과를 구분합니다.

Viewing devices
===============

The Params section holds configured information.  The Current section
holds running statistics.  The Result is printed after a run or after
interruption.  Example::

    /proc/net/pktgen/eth4@0

    Params: count 100000  min_pkt_size: 60  max_pkt_size: 60
        frags: 0  delay: 0  clone_skb: 64  ifname: eth4@0
        flows: 0 flowlen: 0
        queue_map_min: 0  queue_map_max: 0
        dst_min: 192.168.81.2  dst_max:
        src_min:   src_max:
        src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
        udp_src_min: 9  udp_src_max: 109  udp_dst_min: 9  udp_dst_max: 9
        src_mac_count: 0  dst_mac_count: 0
        Flags: UDPSRC_RND  NO_TIMESTAMP  QUEUE_MAP_CPU
    Current:
        pkts-sofar: 100000  errors: 0
        started: 623913381008us  stopped: 623913396439us idle: 25us
        seq_num: 100001  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
        cur_saddr: 192.168.8.3  cur_daddr: 192.168.81.2
        cur_udp_dst: 9  cur_udp_src: 42
        cur_queue_map: 0
        flows: 0
    Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
    6480562pps 3110Mb/sec (3110669760bps) errors: 0

기본 device parameter와 주소 범위

117-166

Device 설정은 `/proc` interface로 하며 sample script에 정의된 `pgset` helper를 쓰는 것이 가장 쉽습니다. `PGDEV` 환경 변수를 대상 procfile로 지정한 뒤 `samples/pktgen/functions.sh`를 source합니다.

`pg_ctrl start`는 packet injection을 시작하고 `pg_ctrl stop`이나 Ctrl-C는 중단합니다. `clone_skb 1`은 같은 packet 복사본 수를 설정하며 `clone_skb 0`은 모든 전송에 단일 SKB를 사용합니다. `burst 8`은 `xmit_more` API로 같은 packet 8개를 queue하고 hardware TX queue tail pointer를 한 번만 갱신합니다. 기본값은 `burst 1`입니다.

`pkt_size`, `frags`, `count`는 packet 크기, fragment 수, 보낼 packet 수를 정합니다. `count 0`은 명시적으로 중단할 때까지 계속 전송합니다. `delay`는 `hard_start_xmit()`에 nanosecond 단위 지연을 추가합니다.

`dst`, `dst_min`, `dst_max`, `src_min`, `src_max`로 IPv4 단일 주소나 순회 범위를 설정합니다. Generator는 매우 공격적으로 traffic을 만들므로 destination을 신중하게 선택해야 합니다. `dst6`와 `src6`는 IPv6 주소, `dstmac`과 `srcmac`은 MAC 주소를 설정합니다.

`queue_map_min`과 `queue_map_max`는 multi-queue device의 TX queue 범위를 정합니다. 특정 queue 1만 선택하려면 둘 다 1로 둡니다. `src_mac_count`와 `dst_mac_count`는 순회할 MAC 수를 정하며 시작 MAC은 각각 `srcmac`과 `dstmac`으로 설정한 값입니다.

기본 pgset parameter
Parameter의미
clone_skb / burstSKB 재사용과 xmit_more batch 크기
pkt_size / frags / countPacket 구조와 전송 개수
delayhard_start_xmit 지연(ns)
dst_min..max / src_min..maxIPv4 주소 범위
dst6 / src6IPv6 주소
dstmac / srcmacMAC 주소
queue_map_min..maxTX queue 선택 범위
src_mac_count / dst_mac_count순회할 MAC 수

Packet 생성량과 주소·queue를 설정합니다.

Configuring devices
===================
This is done via the /proc interface, and most easily done via pgset
as defined in the sample scripts.
You need to specify PGDEV environment variable to use functions from sample
scripts, i.e.::

    export PGDEV=/proc/net/pktgen/eth4@0
    source samples/pktgen/functions.sh

Examples::

 pg_ctrl start           starts injection.
 pg_ctrl stop            aborts injection. Also, ^C aborts generator.

 pgset "clone_skb 1"     sets the number of copies of the same packet
 pgset "clone_skb 0"     use single SKB for all transmits
 pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
                         packet and update HW tx queue tail pointer once.
                         "burst 1" is the default
 pgset "pkt_size 9014"   sets packet size to 9014
 pgset "frags 5"         packet will consist of 5 fragments
 pgset "count 200000"    sets number of packets to send, set to zero
                         for continuous sends until explicitly stopped.

 pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds

 pgset "dst 10.0.0.1"    sets IP destination address
                         (BEWARE! This generator is very aggressive!)

 pgset "dst_min 10.0.0.1"            Same as dst
 pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
 pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
 pgset "src_max 10.0.0.254"          Set the maximum source IP.
 pgset "dst6 fec0::1"     IPV6 destination address
 pgset "src6 fec0::2"     IPV6 source address
 pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
 pgset "srcmac 00:00:00:00:00:00"    sets MAC source address

 pgset "queue_map_min 0" Sets the min value of tx queue interval
 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
                         To select queue 1 of a given device,
                         use queue_map_min=1 and queue_map_max=1

 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
                         The 'minimum' MAC is what you set with srcmac.

 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
                         The 'minimum' MAC is what you set with dstmac.

Random flag, MPLS, VLAN과 rate

167-234

`flag name`은 동작 flag를 켜고 `flag !name`은 끕니다. Interactive shell에서는 `!`가 history expansion으로 처리되지 않도록 작은따옴표를 사용해야 할 수 있습니다.

`IPSRC_RND`, `IPDST_RND`, `UDPSRC_RND`, `UDPDST_RND`, `MACSRC_RND`, `MACDST_RND`, `TXSIZE_RND`는 각 source/destination IP·UDP port·MAC·packet size를 설정 범위에서 무작위로 고릅니다. `IPV6`는 IPv6, `MPLS_RND`는 MPLS label, `VID_RND`와 `SVID_RND`는 VLAN·service VLAN ID 무작위화를 켭니다.

`FLOW_SEQ`는 flow sequence, `QUEUE_MAP_RND`는 무작위 queue map, `QUEUE_MAP_CPU`는 `smp_processor_id()`와 같은 CPU 번호의 queue를 사용합니다. `UDPCSUM`은 UDP checksum을, `IPSEC`은 `CONFIG_XFRM`이 필요한 IPsec encapsulation을, `NODE_ALLOC`은 node-specific memory allocation을 켭니다. `NO_TIMESTAMP`는 timestamp를 끄고 `SHARED`는 shared SKB를 사용합니다.

`spi`는 packet transform에 사용할 특정 Security Association을 정합니다. UDP source와 destination의 min/max는 port 순회 범위를 정의하며 min이 max보다 작으면 범위를 반복합니다.

`mpls`는 comma로 구분한 hexadecimal label stack을 설정합니다. Argument 사이에는 공백이 없어야 하고 leading zero가 필요합니다. Bottom-of-stack bit는 자동으로 설정하므로 직접 넣지 않습니다. 직접 설정하면 해당 주소를 무작위 생성하겠다는 뜻으로 해석되어 `MPLS_RND`가 켜집니다. 고정 label과 무작위 label을 한 stack에 섞을 수 있습니다. `mpls 0` 또는 유효하지 않은 argument는 MPLS를 끕니다.

`vlan_id`, `vlan_p`, `vlan_cfi`와 `svlan_id`, `svlan_p`, `svlan_cfi`는 각각 VLAN 및 service VLAN의 ID(0~4095), priority(0~7), canonical format identifier(0~1)를 설정합니다. `vlan_id 9999`처럼 4095보다 큰 값은 VLAN과 SVLAN tag를 제거하며, `svlan 9999`는 SVLAN tag를 제거합니다.

`tos`는 기존 IPv4 TOS field, `traffic_class`는 기존 IPv6 traffic class를 설정합니다. `rate 300M`은 300Mb/s, `ratep 1000000`은 1Mpps로 rate를 제한합니다.

기본 `xmit_mode`는 `start_xmit`입니다. `xmit_mode netif_receive`는 `netif_receive_skb()`를 통해 RX packet을 stack에 주입합니다. 이 mode는 `burst`와 함께 동작하지만 `clone_skb`와는 함께 쓸 수 없습니다.

Pktgen flag
범주Flag
주소·port 무작위IPSRC_RND, IPDST_RND, UDPSRC_RND, UDPDST_RND, MACSRC_RND, MACDST_RND
크기·label 무작위TXSIZE_RND, MPLS_RND, VID_RND, SVID_RND
Flow·queueFLOW_SEQ, QUEUE_MAP_RND, QUEUE_MAP_CPU
ProtocolIPV6, UDPCSUM, IPSEC
Memory·metadataNODE_ALLOC, NO_TIMESTAMP, SHARED

무작위화와 전송 동작 flag를 묶어 봅니다.

MPLS label 처리
Comma-separated hex labelsLeading zero 유지Bottom-of-stack 자동 설정고정 label stack
Bottom-of-stack bit 직접 설정MPLS_RND 활성화해당 label 무작위 생성

Bottom-of-stack bit 여부가 고정·무작위 label을 구분합니다.

 pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
                         are: IPSRC_RND # IP source is random (between min/max)
                              IPDST_RND # IP destination is random
                              UDPSRC_RND, UDPDST_RND,
                              MACSRC_RND, MACDST_RND
                              TXSIZE_RND, IPV6,
                              MPLS_RND, VID_RND, SVID_RND
                              FLOW_SEQ,
                              QUEUE_MAP_RND # queue map random
                              QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
                              UDPCSUM,
                              IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
                              NODE_ALLOC # node specific memory allocation
                              NO_TIMESTAMP # disable timestamping
                              SHARED # enable shared SKB
 pgset 'flag ![name]'    Clear a flag to determine behaviour.
                         Note that you might need to use single quote in
                         interactive mode, so that your shell wouldn't expand
                         the specified flag as a history command.

 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet.

 pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
                         cycle through the port range.

 pgset "udp_src_max 9"   set UDP source port max.
 pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
                         cycle through the port range.
 pgset "udp_dst_max 9"   set UDP destination port max.

 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
                                         outer label=16,middle label=32,
                                         inner label=0 (IPv4 NULL)) Note that
                                         there must be no spaces between the
                                         arguments. Leading zeros are required.
                                         Do not set the bottom of stack bit,
                                         that's done automatically. If you do
                                         set the bottom of stack bit, that
                                         indicates that you want to randomly
                                         generate that address and the flag
                                         MPLS_RND will be turned on. You
                                         can have any mix of random and fixed
                                         labels in the label stack.

 pgset "mpls 0"                  turn off mpls (or any invalid argument works too!)

 pgset "vlan_id 77"       set VLAN ID 0-4095
 pgset "vlan_p 3"         set priority bit 0-7 (default 0)
 pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)

 pgset "svlan_id 22"      set SVLAN ID 0-4095
 pgset "svlan_p 3"        set priority bit 0-7 (default 0)
 pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)

 pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
 pgset "svlan 9999"       > 4095 remove svlan tag


 pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)

 pgset "rate 300M"        set rate to 300 Mb/s
 pgset "ratep 1000000"    set rate to 1Mpps

 pgset "xmit_mode netif_receive"  RX inject into stack netif_receive_skb()
                                  Works with "burst" but not with "clone_skb".
                                  Default xmit_mode is "start_xmit".

Sample script 사용법

235-270

`samples/pktgen`에는 tutorial script와 helper가 모여 있습니다. `parameters.sh`는 여러 sample script가 parameter를 쉽고 일관되게 parse하도록 지원합니다.

예제는 `pktgen_sample01_simple.sh`에 output interface, destination MAC, destination IP를 전달합니다. `-i` interface는 필수이고 `-s` packet size, `-d` destination IP 또는 CIDR, `-m` destination MAC, `-p` destination port 또는 범위, `-t` thread 수, `-f` 시작 thread index를 지정합니다.

`-c`는 새 SKB를 할당하기 전에 보낼 clone 수, `-n`은 thread당 message 수이며 0은 무한 실행, `-b`는 hardware-level burst입니다. `-v`는 verbose, `-x`는 debug, `-6`은 IPv6, `-w`는 TX delay(ns), `-a`는 기존 generator 상태를 reset하지 않고 설정을 덧붙입니다.

각 option이 설정하는 global variable도 help에 표시됩니다. 예를 들어 필수 `-i`가 `$DEV`를 설정합니다. 자신의 요구에 맞게 `pktgen_sampleXX` script를 복사하여 수정하면 됩니다.

Sample script option
Option설정
-i / -sInterface / packet size
-d / -m / -pDestination IP / MAC / port range
-t / -fThread 수 / 첫 CPU index
-c / -n / -bSKB clone / count / burst
-6 / -wIPv6 / TX delay
-a기존 상태에 설정 추가

자주 쓰는 command-line option입니다.

Sample scripts
==============

A collection of tutorial scripts and helpers for pktgen is in the
samples/pktgen directory. The helper parameters.sh file support easy
and consistent parameter parsing across the sample scripts.

Usage example and help::

 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2

Usage:::

  ./pktgen_sample01_simple.sh [-vx] -i ethX

  -i : ($DEV)       output interface/device (required)
  -s : ($PKT_SIZE)  packet size
  -d : ($DEST_IP)   destination IP. CIDR (e.g. 198.18.0.0/15) is also allowed
  -m : ($DST_MAC)   destination MAC-addr
  -p : ($DST_PORT)  destination PORT range (e.g. 433-444) is also allowed
  -t : ($THREADS)   threads to start
  -f : ($F_THREAD)  index of first thread (zero indexed CPU number)
  -c : ($SKB_CLONE) SKB clones send before alloc new SKB
  -n : ($COUNT)     num messages to send per thread, 0 means indefinitely
  -b : ($BURST)     HW level bursting of SKBs
  -v : ($VERBOSE)   verbose
  -x : ($DEBUG)     debug
  -6 : ($IP6)       IPv6
  -w : ($DELAY)     Tx Delay value (ns)
  -a : ($APPEND)    Script will not reset generator's state, but will append its config

The global variables being set are also listed.  E.g. the required
interface/device parameter "-i" sets variable $DEV.  Copy the
pktgen_sampleXX scripts and modify them to fit your own needs.

Interrupt affinity, IPsec과 shared SKB

271-302

Device를 특정 CPU에 추가할 때 `/proc/irq/XX/smp_affinity`도 설정하여 TX interrupt를 같은 CPU에 묶는 것이 좋습니다. SKB를 해제할 때 CPU 사이 cache bouncing을 줄일 수 있습니다. `QUEUE_MAP_CPU` flag도 실행 thread의 `smp_processor_id()`에 맞는 TX queue로 SKB를 보냅니다.

기본 IPsec transform인 ESP encapsulation과 transport mode는 `flag IPSEC` 및 `flows 1`로 활성화합니다. 기존 AH 또는 tunnel mode testbed script와 호환하려면 `spi SPI_VALUE`로 사용할 transform을 명시합니다.

기본적으로 pktgen이 전송하는 SKB는 user count가 1보다 큰 shared 상태입니다. Non-shared SKB를 시험하려면 `flag !SHARED`로 `SHARED` flag를 제거합니다. 다만 `clone_skb`나 `burst`가 설정되면 pktgen이 나중에 다시 접근할 수 있도록 SKB를 계속 보유해야 하므로 여전히 shared여야 합니다.

CPU locality 최적화
kpktgend_XQUEUE_MAP_CPUTX queue XIRQ smp_affinity CPU XSKB free의 cache bouncing 감소

생성 thread, TX queue와 interrupt를 같은 CPU에 맞춥니다.

Shared SKB 조건
설정SKB 상태
기본SHARED
flag !SHAREDNon-shared 시험
clone_skb 또는 burst 사용후속 접근 때문에 SHARED 필요

Flag를 꺼도 강제로 공유가 필요한 경우가 있습니다.

Interrupt affinity
===================
Note that when adding devices to a specific CPU it is a good idea to
also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
to the same CPU.  This reduces cache bouncing when freeing skbs.

Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
to the running threads CPU (directly from smp_processor_id()).

Enable IPsec
============
Default IPsec transformation with ESP encapsulation plus transport mode
can be enabled by simply setting::

    pgset "flag IPSEC"
    pgset "flows 1"

To avoid breaking existing testbed scripts for using AH type and tunnel mode,
you can use "pgset spi SPI_VALUE" to specify which transformation mode
to employ.

Disable shared SKB
==================
By default, SKBs sent by pktgen are shared (user count > 1).
To test with non-shared SKBs, remove the "SHARED" flag by simply setting::

        pg_set "flag !SHARED"

However, if the "clone_skb" or "burst" parameters are configured, the skb
still needs to be held by pktgen for further access. Hence the skb must be
shared.

전체 command와 configuration option

303-407

Pgcontrol command는 `start`, `stop`, `reset`입니다. Thread command는 `add_device`와 `rem_device_all`입니다.

Device 기본 command에는 `count`, `clone_skb`, `burst`, `debug`, `frags`, `delay`, source/destination MAC count, packet size 및 min/max packet size가 있습니다. Queue 관련 command는 `queue_map_min`, `queue_map_max`, `skb_priority`입니다.

Protocol field에는 IPv4 `tos`, IPv6 `traffic_class`, `mpls`, UDP source/destination min/max, NUMA `node`가 있습니다. `flag` 아래에는 모든 random·IPv6·MPLS·VLAN·flow·queue·checksum·IPsec·allocation·timestamp·shared option이 나열됩니다.

IPsec `spi`, IPv4 destination/source min/max, destination/source MAC, `clear_counters`, IPv6 source/destination 및 destination min/max, `flows`, `flowlen`, bit rate `rate`, packet rate `ratep`도 지원합니다.

`xmit_mode`는 `start_xmit` 또는 `netif_receive`를 선택합니다. VLAN과 service VLAN에는 각각 CFI, ID, priority command가 있습니다. 이 목록은 앞 절의 사용 예를 한곳에 모은 reference입니다.

Command group
범위주요 command
Pgcontrolstart, stop, reset
Threadadd_device, rem_device_all
Packetcount, clone_skb, burst, frags, delay, pkt_size
Queuequeue_map_min/max, skb_priority
Addresssrc/dst IPv4·IPv6·MAC, UDP port ranges
Protocoltos, traffic_class, MPLS, VLAN, SVLAN, IPsec SPI
Flow·rateflows, flowlen, rate, ratep, xmit_mode

Procfile 범위별 명령을 정리합니다.

Current commands and configuration options
==========================================

**Pgcontrol commands**::

    start
    stop
    reset

**Thread commands**::

    add_device
    rem_device_all


**Device commands**::

    count
    clone_skb
    burst
    debug

    frags
    delay

    src_mac_count
    dst_mac_count

    pkt_size
    min_pkt_size
    max_pkt_size

    queue_map_min
    queue_map_max
    skb_priority

    tos           (ipv4)
    traffic_class (ipv6)

    mpls

    udp_src_min
    udp_src_max

    udp_dst_min
    udp_dst_max

    node

    flag
    IPSRC_RND
    IPDST_RND
    UDPSRC_RND
    UDPDST_RND
    MACSRC_RND
    MACDST_RND
    TXSIZE_RND
    IPV6
    MPLS_RND
    VID_RND
    SVID_RND
    FLOW_SEQ
    QUEUE_MAP_RND
    QUEUE_MAP_CPU
    UDPCSUM
    IPSEC
    NODE_ALLOC
    NO_TIMESTAMP
    SHARED

    spi (ipsec)

    dst_min
    dst_max

    src_min
    src_max

    dst_mac
    src_mac

    clear_counters

    src6
    dst6
    dst6_max
    dst6_min

    flows
    flowlen

    rate
    ratep

    xmit_mode <start_xmit|netif_receive>

    vlan_cfi
    vlan_id
    vlan_p

    svlan_cfi
    svlan_id
    svlan_p

참고 자료와 기여자

408-422

원문은 SLU의 pktgen testing FTP archive와 examples archive를 참고 자료로 제공합니다. 2004년 Erlangen Linux-Kongress에서 발표된 pktgen paper PDF도 같은 archive에 있습니다. URL은 원문에 그대로 보존됩니다.

IA-64와 parisc에서 시험한 Grant Grundler를 비롯해 Harald Welte, Lennert Buytenhek, Stephen Hemminger, Andi Kleen, Dave Miller 및 많은 기여자에게 감사를 표합니다. 문서는 Linux network development의 성공을 기원하며 끝납니다.

References:

- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/

Paper from Linux-Kongress in Erlangen 2004.
- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf

Thanks to:

Grant Grundler for testing on IA-64 and parisc, Harald Welte,  Lennert Buytenhek
Stephen Hemminger, Andi Kleen, Dave Miller and many others.


Good luck with the linux net-development.