CoreSight - Perf

Perf 기록·decode와 OpenCSD build

이 문서는 Carsten Haitzler가 2022년 6월 29일에 작성했으며, Perf가 CoreSight trace data를 로컬에서 수집하고 해석하는 방법을 설명한다.

Perf는 CoreSight trace data를 직접 읽어 output perf data file에 저장한다. 이후 이 data를 decode하면 debugging 또는 profiling을 위해 trace된 instruction을 확인할 수 있다.

`perf record -e cs_etm//u testbinary`는 `testbinary`가 종료될 때까지 실행하면서 `perf.data` trace file을 만든다. CoreSight가 정상 동작하면 이 file에 AUX section이 포함된다.

`perf report --stdio --dump -i perf.data`는 file 내용을 읽을 수 있는 text로 dump한다. 정상 AUX block에는 `PERF_RECORD_AUXTRACE`의 size, offset, reference, index, TID, CPU와 함께 CoreSight ETM packet이 나타난다.

예제 packet은 alignment synchronization인 `I_ASYNC`, trace 정보 `I_TRACE_INFO`, 64-bit address `I_ADDR_L_64IS0`, `I_TRACE_ON`, AArch64 EL0 non-secure context를 포함한 `I_ADDR_CTXT_L_64IS0`, 여러 `I_ATOM_F6` atom을 보여 준다. 이런 내용이 보이면 CoreSight data tracing이 정상이다.

`tools/perf`에서 CoreSight 지원을 넣어 Perf를 build하려면 `make CORESIGHT=1`을 사용한다. build에는 OpenCSD가 필요하며 `libopencsd`, `libopencsd-dev` distribution package를 설치하거나 upstream source를 내려받아 직접 build할 수 있다.

upstream OpenCSD는 `https://github.com/Linaro/OpenCSD`에 있고, Perf CoreSight build와 확장 사용법은 `https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md`를 참조한다.

.. SPDX-License-Identifier: GPL-2.0

================
CoreSight - Perf
================

    :Author:   Carsten Haitzler <[email protected]>
    :Date:     June 29th, 2022

Perf is able to locally access CoreSight trace data and store it to the
output perf data files. This data can then be later decoded to give the
instructions that were traced for debugging or profiling purposes. You
can log such data with a perf record command like::

   perf record -e cs_etm//u testbinary

This would run some test binary (testbinary) until it exits and record
a perf.data trace file. That file would have AUX sections if CoreSight
is working correctly. You can dump the content of this file as
readable text with a command like::

   perf report --stdio --dump -i perf.data

You should find some sections of this file have AUX data blocks like::

   0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1

   . ... CoreSight ETM Trace data: size 73168 bytes
           Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
             Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
             Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
             Idx:26; ID:10;  I_TRACE_ON : Trace On.
             Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
             Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
             Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
             Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
             Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
             ...

If you see these above, then your system is tracing CoreSight data
correctly.

To compile perf with CoreSight support in the tools/perf directory do::

    make CORESIGHT=1

This requires OpenCSD to build. You may install distribution packages
for the support such as libopencsd and libopencsd-dev or download it
and build yourself. Upstream OpenCSD is located at:

  https://github.com/Linaro/OpenCSD

For complete information on building perf with CoreSight support and
more extensive usage look at:

  https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md

Kernel CoreSight configuration

kernel configuration에서도 CoreSight 지원을 enable해야 하며 기본 option은 `CONFIG_CORESIGHT=y`다.

일반적으로 link와 sink, TMC, CATU, TPIU, ETB v1.0, ETMv4 source, CTI와 CTI integration register option도 함께 enable한다. 각 option의 세부 조건은 kernel configuration help를 참조한다.

Kernel CoreSight Support
------------------------

You will also want CoreSight support enabled in your kernel config.
Ensure it is enabled with::

   CONFIG_CORESIGHT=y

There are various other CoreSight options you probably also want
enabled like::

   CONFIG_CORESIGHT_LINKS_AND_SINKS=y
   CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
   CONFIG_CORESIGHT_CATU=y
   CONFIG_CORESIGHT_SINK_TPIU=y
   CONFIG_CORESIGHT_SINK_ETBV10=y
   CONFIG_CORESIGHT_SOURCE_ETM4X=y
   CONFIG_CORESIGHT_CTI=y
   CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y

Please refer to the kernel configuration help for more information.

Event 기반 AUX pause와 resume

Arm CoreSight는 매우 많은 hardware trace data를 만들 수 있어 recording overhead를 높이고 profiling 결과 검토를 방해할 수 있다. Perf의 AUX pause/resume 기능은 필요한 구간만 세밀하게 trace해 과도한 data를 줄인다.

pause와 resume은 연계 event로 trigger할 수 있다. event는 static/dynamic ftrace tracepoint 또는 CPU PMU cycle event 같은 PMU event가 될 수 있다.

`aux-action=start-paused`는 `cs_etm` PMU event를 paused state로 시작한다. `aux-action=pause`가 붙은 연계 event는 AUX trace를 멈추고, `aux-action=resume`이 붙은 event는 다시 시작한다.

ftrace 예제는 `sys_enter_openat`에서 resume하고 `sys_exit_openat`에서 pause해 `ls`의 openat 구간을 기록한다. PMU 예제는 system-wide recording에서 cycle event의 서로 다른 period로 pause와 resume을 반복하며 1초 동안 실행한다.

Fine-grained tracing with AUX pause and resume
----------------------------------------------

Arm CoreSight may generate a large amount of hardware trace data, which
will lead to overhead in recording and distract users when reviewing
profiling result. To mitigate the issue of excessive trace data, Perf
provides AUX pause and resume functionality for fine-grained tracing.

The AUX pause and resume can be triggered by associated events. These
events can be ftrace tracepoints (including static and dynamic
tracepoints) or PMU events (e.g. CPU PMU cycle event). To create a perf
session with AUX pause / resume, three configuration terms are
introduced:

- "aux-action=start-paused": it is specified for the cs_etm PMU event to
  launch in a paused state.
- "aux-action=pause": an associated event is specified with this term
  to pause AUX trace.
- "aux-action=resume": an associated event is specified with this term
  to resume AUX trace.

Example for triggering AUX pause and resume with ftrace tracepoints::

  perf record -e cs_etm/aux-action=start-paused/k,syscalls:sys_enter_openat/aux-action=resume/,syscalls:sys_exit_openat/aux-action=pause/ ls

Example for triggering AUX pause and resume with PMU event::

  perf record -a -e cs_etm/aux-action=start-paused/k \
        -e cycles/aux-action=pause,period=10000000/ \
        -e cycles/aux-action=resume,period=1050000/ -- sleep 1

Perf CoreSight self-test와 통계 변동

`perf test`는 많은 self-test를 실행하며 CoreSight가 enable된 ARM64에서는 CoreSight test도 포함한다. 일반적으로 kernel tree의 `tools/perf` directory에서 실행한다.

일부 test는 `Check Arm CoreSight trace data recording and synthesized samples`, `Check Arm SPE trace data recording and synthesized samples`처럼 Perf 내부 지원을 확인한다.

다른 test는 `tests/shell/coresight`의 test binary에 실제 `perf record`를 실행해 최소 기능 수준을 확인한다. 예로 ASM pure loop, 16 KB memcpy 10 thread, TID를 확인하는 10-thread loop 등이 있으며 실행 script도 같은 directory에 있다.

`tests/shell/coresight/*/` 아래 tool binary가 없으면 이 recording test는 실행되지 않고 skip된다. hardware에 CoreSight가 없다면 Perf를 CoreSight support 없이 build하거나 해당 binary를 제거해 실패 대신 skip되게 해야 한다.

test는 현재 working directory, 예를 들어 `tools/perf`에 `stats-*.csv` 이름으로 이전 결과를 누적한다. `stats-asm_pure_loop-out.csv`, `stats-memcpy_thread-16k_10.csv`가 예다.

CSV는 perf data의 AUX section에서 특정 encoding 수를 세어 단순한 동작 지표를 남긴다. CoreSight는 data 양이 충분히 크면 processor가 제때 깨어 buffer를 모두 읽지 못해 일부 trace를 잃을 수 있다. 그래서 `perf test` 실행마다 수집량이 크게 달라질 수 있다.

test를 여러 번 실행하면 CSV에 data가 계속 append되므로 시간에 따른 변화를 검토하거나 graph로 만들어 품질이 좋아졌는지 나빠졌는지 판단할 수 있다.

필요한 data를 모두 capture하지 못해 test가 가끔 실패할 수 있다. 이 test의 목적은 생성되는 trace의 품질과 양을 장기 추적하고 Linux kernel 변경이 품질을 개선하는 시점을 찾는 데 있다.

일부 test는 perf data file을 처리하고 내용을 dump해 검사하는 단계 때문에 상당히 오래 걸릴 수 있다.

Perf test - Verify kernel and userspace perf CoreSight work
-----------------------------------------------------------

When you run perf test, it will do a lot of self tests. Some of those
tests will cover CoreSight (only if enabled and on ARM64). You
generally would run perf test from the tools/perf directory in the
kernel tree. Some tests will check some internal perf support like:

   Check Arm CoreSight trace data recording and synthesized samples
   Check Arm SPE trace data recording and synthesized samples

Some others will actually use perf record and some test binaries that
are in tests/shell/coresight and will collect traces to ensure a
minimum level of functionality is met. The scripts that launch these
tests are in the same directory. These will all look like:

   CoreSight / ASM Pure Loop
   CoreSight / Memcpy 16k 10 Threads
   CoreSight / Thread Loop 10 Threads - Check TID
   etc.

These perf record tests will not run if the tool binaries do not exist
in tests/shell/coresight/\*/ and will be skipped. If you do not have
CoreSight support in hardware then either do not build perf with
CoreSight support or remove these binaries in order to not have these
tests fail and have them skip instead.

These tests will log historical results in the current working
directory (e.g. tools/perf) and will be named stats-\*.csv like:

   stats-asm_pure_loop-out.csv
   stats-memcpy_thread-16k_10.csv
   ...

These statistic files log some aspects of the AUX data sections in
the perf data output counting some numbers of certain encodings (a
good way to know that it's working in a very simple way). One problem
with CoreSight is that given a large enough amount of data needing to
be logged, some of it can be lost due to the processor not waking up
in time to read out all the data from buffers etc.. You will notice
that the amount of data collected can vary a lot per run of perf test.
If you wish to see how this changes over time, simply run perf test
multiple times and all these csv files will have more and more data
appended to it that you can later examine, graph and otherwise use to
figure out if things have become worse or better.

This means sometimes these tests fail as they don't capture all the
data needed. This is about tracking quality and amount of data
produced over time and to see when changes to the Linux kernel improve
quality of traces.

Be aware that some of these tests take quite a while to run, specifically
in processing the perf data file and dumping contents to then examine what
is inside.

Test output 장기 보관 경로

`PERF_TEST_CORESIGHT_STATDIR` environment variable을 `perf test` 전에 설정하면 CSV log 저장 위치를 바꿀 수 있다. 원문 예제는 `/var/tmp`로 지정한다.

test는 나중에 검사할 Perf output data도 현재 directory에 남긴다. `perf-asm_pure_loop-out.data`, `perf-memcpy_thread-16k_10.data`가 예다.

`PERF_TEST_CORESIGHT_DATADIR` environment variable은 perf data file 저장 위치를 바꾼다. CSV와 perf data를 현재 working directory 밖에서 장기간 보관하고 검토하려면 두 environment variable을 설정하는 것이 좋다.

You can change where these csv logs are stored by setting the
PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
test like::

   export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
   perf test

They will also store resulting perf output data in the current
directory for later inspection like::

   perf-asm_pure_loop-out.data
   perf-memcpy_thread-16k_10.data
   ...

You can alter where the perf data files are stored by setting the
PERF_TEST_CORESIGHT_DATADIR environment variable such as::

   PERF_TEST_CORESIGHT_DATADIR=/var/tmp
   perf test

You may wish to set these above environment variables if you wish to
keep the output of tests outside of the current working directory for
longer term storage and examination.