CoreSight - HW Assisted Tracing on ARM

Source·link·sink와 APB·ATB·CTM 구조

이 문서는 Mathieu Poirier가 2014년 9월 11일에 작성했다. CoreSight는 ARM 기반 SoC debugging을 위한 기술군으로 JTAG와 hardware-assisted tracing을 포함하며, 이 문서는 후자를 다룬다.

여러 SoC와 GPU, DMA engine 같은 구성요소가 있는 system에서는 hardware-assisted tracing의 가치가 커진다. ARM은 synthesis 단계에서 특정 tracing 요구에 맞는 구성요소를 design에 추가하는 방식으로 이 기능을 제공한다. 구성요소는 보통 source, link, sink로 분류되고 AMBA bus를 통해 발견된다.

source는 사용자가 설정한 tracing scenario에 따라 processor instruction path를 나타내는 compressed stream을 만든다. stream은 ATB bus를 통해 source와 sink를 잇는 link를 지나간다.

sink는 CoreSight 구현의 endpoint다. compressed stream을 on-board memory buffer에 저장하거나 외부 interface를 만들어 data를 host로 보낸다. 외부 전송 sink를 사용하면 on-board CoreSight memory buffer가 가득 찰 위험 없이 장시간 수집할 수 있다.

원문 시스템 그림에는 CPU별 ETM/PTM source, STM source, CTI, funnel, replicator, ETB, TPIU, system memory, DAP와 SWD/JTAG가 함께 나타난다. AMBA AXI는 system memory 쪽 data path를, Debug APB는 target 내부 programming을, ATB는 trace stream을 운반한다.

target에서 구성요소 설정은 APB bus로 수행하지만 trace data는 별도의 ATB bus로 전달된다. CTM은 CoreSight 구성요소 사이의 trigger signal을 모으고 분배한다.

CoreSight framework는 platform의 CoreSight device를 표현·설정·관리하는 중앙 지점이다. 초기 구현은 ETM/PTM, funnel, replicator, TMC, TPIU, ETB의 기본 tracing 기능에 초점을 맞췄으며 문서 작성 당시 STM과 CTI 같은 복잡한 IP block은 후속 작업으로 설명됐다.

======================================
Coresight - HW Assisted Tracing on ARM
======================================

   :Author:   Mathieu Poirier <[email protected]>
   :Date:     September 11th, 2014

Introduction
------------

Coresight is an umbrella of technologies allowing for the debugging of ARM
based SoC.  It includes solutions for JTAG and HW assisted tracing.  This
document is concerned with the latter.

HW assisted tracing is becoming increasingly useful when dealing with systems
that have many SoCs and other components like GPU and DMA engines.  ARM has
developed a HW assisted tracing solution by means of different components, each
being added to a design at synthesis time to cater to specific tracing needs.
Components are generally categorised as source, link and sinks and are
(usually) discovered using the AMBA bus.

"Sources" generate a compressed stream representing the processor instruction
path based on tracing scenarios as configured by users.  From there the stream
flows through the coresight system (via ATB bus) using links that are connecting
the emanating source to a sink(s).  Sinks serve as endpoints to the coresight
implementation, either storing the compressed stream in a memory buffer or
creating an interface to the outside world where data can be transferred to a
host without fear of filling up the onboard coresight memory buffer.

At typical coresight system would look like this::

  *****************************************************************
 **************************** AMBA AXI  ****************************===||
  *****************************************************************    ||
        ^                    ^                            |            ||
        |                    |                            *            **
     0000000    :::::     0000000    :::::    :::::    @@@@@@@    ||||||||||||
     0 CPU 0<-->: C :     0 CPU 0<-->: C :    : C :    @ STM @    || System ||
  |->0000000    : T :  |->0000000    : T :    : T :<--->@@@@@     || Memory ||
  |  #######<-->: I :  |  #######<-->: I :    : I :      @@@<-|   ||||||||||||
  |  # ETM #    :::::  |  # PTM #    :::::    :::::       @   |
  |   #####      ^ ^   |   #####      ^ !      ^ !        .   |   |||||||||
  | |->###       | !   | |->###       | !      | !        .   |   || DAP ||
  | |   #        | !   | |   #        | !      | !        .   |   |||||||||
  | |   .        | !   | |   .        | !      | !        .   |      |  |
  | |   .        | !   | |   .        | !      | !        .   |      |  *
  | |   .        | !   | |   .        | !      | !        .   |      | SWD/
  | |   .        | !   | |   .        | !      | !        .   |      | JTAG
  *****************************************************************<-|
 *************************** AMBA Debug APB ************************
  *****************************************************************
   |    .          !         .          !        !        .    |
   |    .          *         .          *        *        .    |
  *****************************************************************
 ******************** Cross Trigger Matrix (CTM) *******************
  *****************************************************************
   |    .     ^              .                            .    |
   |    *     !              *                            *    |
  *****************************************************************
 ****************** AMBA Advanced Trace Bus (ATB) ******************
  *****************************************************************
   |          !                        ===============         |
   |          *                         ===== F =====<---------|
   |   :::::::::                         ==== U ====
   |-->:: CTI ::<!!                       === N ===
   |   :::::::::  !                        == N ==
   |    ^         *                        == E ==
   |    !  &&&&&&&&&       IIIIIII         == L ==
   |------>&& ETB &&<......II     I        =======
   |    !  &&&&&&&&&       II     I           .
   |    !                    I     I          .
   |    !                    I REP I<..........
   |    !                    I     I
   |    !!>&&&&&&&&&       II     I           *Source: ARM ltd.
   |------>& TPIU  &<......II    I            DAP = Debug Access Port
           &&&&&&&&&       IIIIIII            ETM = Embedded Trace Macrocell
               ;                              PTM = Program Trace Macrocell
               ;                              CTI = Cross Trigger Interface
               *                              ETB = Embedded Trace Buffer
          To trace port                       TPIU= Trace Port Interface Unit
                                              SWD = Serial Wire Debug

While on target configuration of the components is done via the APB bus,
all trace data are carried out-of-band on the ATB bus.  The CTM provides
a way to aggregate and distribute signals between CoreSight components.

The coresight framework provides a central point to represent, configure and
manage coresight devices on a platform.  This first implementation centers on
the basic tracing functionality, enabling components such ETM/PTM, funnel,
replicator, TMC, TPIU and ETB.  Future work will enable more
intricate IP blocks such as STM and CTI.

약어·분류와 Device Tree binding

PTM은 Program Trace Macrocell, ETM은 Embedded Trace Macrocell, STM은 System Trace Macrocell, ETB는 Embedded Trace Buffer, ITM은 Instrumentation Trace Macrocell이다.

TPIU는 Trace Port Interface Unit이다. TMC-ETR은 Embedded Trace Router로 설정된 Trace Memory Controller이고, TMC-ETF는 Embedded Trace FIFO로 설정된 Trace Memory Controller다. CTI는 Cross Trigger Interface다.

source class에는 ETMv3.x, ETMv4, PTMv1.0, PTMv1.1, STM, STM500, ITM이 있다. link에는 intelligent 여부와 관계없는 funnel, replicator와 TMC-ETR이 들어간다. sink에는 ETBv1.0, ETB1.1, TPIU, TMC-ETF가 있고, CTI는 misc로 분류된다.

Device Tree binding 세부 사항은 `Documentation/devicetree/bindings/arm/arm,coresight-*.yaml`을 참조한다.

문서 작성 당시 ITM, STM, CTI driver는 제공되지 않았고 solution이 성숙하면서 추가될 것으로 예상했다. 이는 문서의 작성 시점 설명이며 현재 source와 page의 원문은 그대로 보존한다.

Acronyms and Classification
---------------------------

Acronyms:

PTM:
    Program Trace Macrocell
ETM:
    Embedded Trace Macrocell
STM:
    System trace Macrocell
ETB:
    Embedded Trace Buffer
ITM:
    Instrumentation Trace Macrocell
TPIU:
     Trace Port Interface Unit
TMC-ETR:
        Trace Memory Controller, configured as Embedded Trace Router
TMC-ETF:
        Trace Memory Controller, configured as Embedded Trace FIFO
CTI:
    Cross Trigger Interface

Classification:

Source:
   ETMv3.x ETMv4, PTMv1.0, PTMv1.1, STM, STM500, ITM
Link:
   Funnel, replicator (intelligent or not), TMC-ETR
Sinks:
   ETBv1.0, ETB1.1, TPIU, TMC-ETF
Misc:
   CTI


Device Tree Bindings
--------------------

See ``Documentation/devicetree/bindings/arm/arm,coresight-*.yaml`` for details.

As of this writing drivers for ITM, STMs and CTIs are not provided but are
expected to be added as the solution matures.

Framework API·descriptor와 device naming

CoreSight compliant device는 올바른 API를 사용하면 framework에 등록할 수 있다. `coresight_register(struct coresight_desc *desc)`는 descriptor를 받아 device를 core framework에 등록하고 `struct coresight_device *`를 반환한다. `coresight_unregister(struct coresight_device *csdev)`는 등록 때 얻은 device reference로 등록을 해제한다.

등록이 성공하면 device는 `/sys/bus/coresight/devices` 아래에 나타난다. TC2 예제에는 replicator, TPIU, ETB, funnel, PTM과 ETM device가 보인다.

`struct coresight_desc`에는 `enum coresight_dev_type type`, `struct coresight_dev_subtype subtype`, `const struct coresight_ops *ops`, `struct coresight_platform_data *pdata`, `struct device *dev`, `const struct attribute_group **groups`가 들어간다.

`coresight_dev_type`은 source, link, sink 중 device의 기본 class를 식별하고 `coresight_dev_subtype`은 그 type을 더 세분한다.

`struct coresight_ops`는 필수이며 framework가 구성요소의 기본 operation을 수행하는 방법을 제공한다. class마다 요구가 달라 `struct coresight_ops_sink`, `struct coresight_ops_link`, `struct coresight_ops_source`가 별도로 제공된다.

driver probe에서 `of_get_coresight_platform_data()`로 `pdata`를 얻는다. `dev`에는 `amba_device`에 포함된 device reference를 넣으며 예제의 `etm_probe()`는 `drvdata->dev = &adev->dev`로 설정한다.

source, link, sink class에는 `struct coresight_ops`가 정의한 generic operation이 있다. `groups`는 해당 구성요소에만 고유한 operation을 위한 sysfs entry 목록이며 implementation-defined customization은 이 entry로 접근하고 제어한다.

초기 device 이름은 AMBA/platform bus의 실제 parent device 이름과 같았다. Linux Open Firmware naming에 따라 base physical address 뒤에 device type을 붙여 `20010000.etf`, `20040000.funnel`, `22040000.etm`처럼 만들었다.

ACPI 지원이 도입되면서 실제 device 이름이 난해해져 type 기반 generic naming scheme이 추가됐다. CPU에 binding된 device는 CPU logical number를 사용하므로 CPU0 ETM은 `etm0`이다.

그 밖의 device는 `<device_type_prefix>N` 형식이다. prefix는 device type에 고유하고 N은 probe 순서에 따라 부여되는 연속 번호다. 예는 `tmc_etf0`, `tmc_etr0`, `funnel0`, `funnel1`이다.

이후 예제에는 old naming과 new naming이 모두 나타날 수 있다. 실제 작업에서는 지정된 system location 아래에 표시되는 이름을 그대로 사용해야 한다.

Framework and implementation
----------------------------

The coresight framework provides a central point to represent, configure and
manage coresight devices on a platform.  Any coresight compliant device can
register with the framework for as long as they use the right APIs:

.. c:function:: struct coresight_device *coresight_register(struct coresight_desc *desc);
.. c:function:: void coresight_unregister(struct coresight_device *csdev);

The registering function is taking a ``struct coresight_desc *desc`` and
register the device with the core framework. The unregister function takes
a reference to a ``struct coresight_device *csdev`` obtained at registration time.

If everything goes well during the registration process the new devices will
show up under /sys/bus/coresight/devices, as showns here for a TC2 platform::

    root:~# ls /sys/bus/coresight/devices/
    replicator  20030000.tpiu    2201c000.ptm  2203c000.etm  2203e000.etm
    20010000.etb         20040000.funnel  2201d000.ptm  2203d000.etm
    root:~#

The functions take a ``struct coresight_device``, which looks like this::

    struct coresight_desc {
            enum coresight_dev_type type;
            struct coresight_dev_subtype subtype;
            const struct coresight_ops *ops;
            struct coresight_platform_data *pdata;
            struct device *dev;
            const struct attribute_group **groups;
    };


The "coresight_dev_type" identifies what the device is, i.e, source link or
sink while the "coresight_dev_subtype" will characterise that type further.

The ``struct coresight_ops`` is mandatory and will tell the framework how to
perform base operations related to the components, each component having
a different set of requirement. For that ``struct coresight_ops_sink``,
``struct coresight_ops_link`` and ``struct coresight_ops_source`` have been
provided.

The next field ``struct coresight_platform_data *pdata`` is acquired by calling
``of_get_coresight_platform_data()``, as part of the driver's _probe routine and
``struct device *dev`` gets the device reference embedded in the ``amba_device``::

    static int etm_probe(struct amba_device *adev, const struct amba_id *id)
    {
     ...
     ...
     drvdata->dev = &adev->dev;
     ...
    }

Specific class of device (source, link, or sink) have generic operations
that can be performed on them (see ``struct coresight_ops``). The ``**groups``
is a list of sysfs entries pertaining to operations
specific to that component only.  "Implementation defined" customisations are
expected to be accessed and controlled using those entries.

Device Naming scheme
--------------------

The devices that appear on the "coresight" bus were named the same as their
parent devices, i.e, the real devices that appears on AMBA bus or the platform bus.
Thus the names were based on the Linux Open Firmware layer naming convention,
which follows the base physical address of the device followed by the device
type. e.g::

    root:~# ls /sys/bus/coresight/devices/
     20010000.etf  20040000.funnel      20100000.stm     22040000.etm
     22140000.etm  230c0000.funnel      23240000.etm     20030000.tpiu
     20070000.etr  20120000.replicator  220c0000.funnel
     23040000.etm  23140000.etm         23340000.etm

However, with the introduction of ACPI support, the names of the real
devices are a bit cryptic and non-obvious. Thus, a new naming scheme was
introduced to use more generic names based on the type of the device. The
following rules apply::

  1) Devices that are bound to CPUs, are named based on the CPU logical
     number.

     e.g, ETM bound to CPU0 is named "etm0"

  2) All other devices follow a pattern, "<device_type_prefix>N", where :

	<device_type_prefix> 	- A prefix specific to the type of the device
	N			- a sequential number assigned based on the order
				  of probing.

	e.g, tmc_etf0, tmc_etr0, funnel0, funnel1

Thus, with the new scheme the devices could appear as ::

    root:~# ls /sys/bus/coresight/devices/
     etm0     etm1     etm2         etm3  etm4      etm5      funnel0
     funnel1  funnel2  replicator0  stm0  tmc_etf0  tmc_etr0  tpiu0

Some of the examples below might refer to old naming scheme and some
to the newer scheme, to give a confirmation that what you see on your
system is not unexpected. One must use the "names" as they appear on
the system under specified locations.

connections directory로 trace topology 탐색

각 CoreSight component의 `connections` directory에는 다른 CoreSight component로 가는 link가 있다. 이를 통해 trace topology를 탐색하고 큰 system에서 특정 source에 가장 적절한 sink를 정할 수 있다. CTI가 어느 component에 연결됐는지도 확인할 수 있으며 `nr_links`는 directory의 link 수를 나타낸다.

Juno platform의 `etm0` 예제에서 `connections`는 `cti_cpu0` 연결과 output port 0에서 `funnel2`로 가는 link를 보여 준다.

`funnel2`는 `etm0`, `etm3`, `etm4`, `etm5`를 input 0-3으로 받아 output 0으로 `funnel0`에 보낸다. `funnel0`은 `funnel1`과 `funnel2`를 받아 `tmc_etf0`으로 보낸다.

첫 sink인 `tmc_etf0`은 data를 직접 수집하는 sink로도, 뒤쪽 chain으로 전달하는 link로도 사용할 수 있다. 예제에서는 `cti_sys0`과 연결되고 `funnel0`에서 입력받아 `funnel4`로 출력한다.

`funnel4`는 `tmc_etf0`과 `tmc_etf1`을 합쳐 `replicator0`으로 보낸다. `replicator0`은 stream을 output 0의 `tpiu0`과 output 1의 `tmc_etr0`으로 복제한다. 최종 sink `tmc_etr0`은 `cti_sys0`과 연결되고 replicator input을 받는다.

sysfs를 사용할 때는 sink와 source만 enable하면 된다. framework가 필요한 모든 intermediate link를 올바르게 enable한다.

`cti_sys0`은 여러 connections 목록에 나타난다. CTI는 여러 device에 연결할 수 있고 CTM을 중심으로 star topology를 이룬다. 예제의 `cti_sys0`은 STM0, TMC-ETF0, TMC-ETR0, TPIU0 네 component와 연결된다. 자세한 내용은 `Documentation/trace/coresight/coresight-ect.rst`를 참조한다.

Topology Representation
-----------------------

Each CoreSight component has a ``connections`` directory which will contain
links to other CoreSight components. This allows the user to explore the trace
topology and for larger systems, determine the most appropriate sink for a
given source. The connection information can also be used to establish
which CTI devices are connected to a given component. This directory contains a
``nr_links`` attribute detailing the number of links in the directory.

For an ETM source, in this case ``etm0`` on a Juno platform, a typical
arrangement will be::

  linaro-developer:~# ls - l /sys/bus/coresight/devices/etm0/connections
  <file details>  cti_cpu0 -> ../../../23020000.cti/cti_cpu0
  <file details>  nr_links
  <file details>  out:0 -> ../../../230c0000.funnel/funnel2

Following the out port to ``funnel2``::

  linaro-developer:~# ls -l /sys/bus/coresight/devices/funnel2/connections
  <file details> in:0 -> ../../../23040000.etm/etm0
  <file details> in:1 -> ../../../23140000.etm/etm3
  <file details> in:2 -> ../../../23240000.etm/etm4
  <file details> in:3 -> ../../../23340000.etm/etm5
  <file details> nr_links
  <file details> out:0 -> ../../../20040000.funnel/funnel0

And again to ``funnel0``::

  linaro-developer:~# ls -l /sys/bus/coresight/devices/funnel0/connections
  <file details> in:0 -> ../../../220c0000.funnel/funnel1
  <file details> in:1 -> ../../../230c0000.funnel/funnel2
  <file details> nr_links
  <file details> out:0 -> ../../../20010000.etf/tmc_etf0

Finding the first sink ``tmc_etf0``. This can be used to collect data
as a sink, or as a link to propagate further along the chain::

  linaro-developer:~# ls -l /sys/bus/coresight/devices/tmc_etf0/connections
  <file details> cti_sys0 -> ../../../20020000.cti/cti_sys0
  <file details> in:0 -> ../../../20040000.funnel/funnel0
  <file details> nr_links
  <file details> out:0 -> ../../../20150000.funnel/funnel4

via ``funnel4``::

  linaro-developer:~# ls -l /sys/bus/coresight/devices/funnel4/connections
  <file details> in:0 -> ../../../20010000.etf/tmc_etf0
  <file details> in:1 -> ../../../20140000.etf/tmc_etf1
  <file details> nr_links
  <file details> out:0 -> ../../../20120000.replicator/replicator0

and a ``replicator0``::

  linaro-developer:~# ls -l /sys/bus/coresight/devices/replicator0/connections
  <file details> in:0 -> ../../../20150000.funnel/funnel4
  <file details> nr_links
  <file details> out:0 -> ../../../20030000.tpiu/tpiu0
  <file details> out:1 -> ../../../20070000.etr/tmc_etr0

Arriving at the final sink in the chain, ``tmc_etr0``::

  linaro-developer:~# ls -l /sys/bus/coresight/devices/tmc_etr0/connections
  <file details> cti_sys0 -> ../../../20020000.cti/cti_sys0
  <file details> in:0 -> ../../../20120000.replicator/replicator0
  <file details> nr_links

As described below, when using sysfs it is sufficient to enable a sink and
a source for successful trace. The framework will correctly enable all
intermediate links as required.

Note: ``cti_sys0`` appears in two of the connections lists above.
CTIs can connect to multiple devices and are arranged in a star topology
via the CTM. See (Documentation/trace/coresight/coresight-ect.rst)
[#fourth]_ for further details.
Looking at this device we see 4 connections::

  linaro-developer:~# ls -l /sys/bus/coresight/devices/cti_sys0/connections
  <file details> nr_links
  <file details> stm0 -> ../../../20100000.stm/stm0
  <file details> tmc_etf0 -> ../../../20010000.etf/tmc_etf0
  <file details> tmc_etr0 -> ../../../20070000.etr/tmc_etr0
  <file details> tpiu0 -> ../../../20030000.tpiu/tpiu0

sysfs로 sink·source enable과 ETB 수집

CoreSight framework는 Perf command-line tool 또는 CoreSight device sysfs interface 두 방식으로 사용할 수 있다. sysfs 방식은 CoreSight hardware에 대한 깊은 이해가 필요하므로 Perf 방식이 권장된다.

trace 수집 전에 sink를 선택해야 한다. 동시에 enable할 수 있는 sink나 source 수에는 제한이 없다. sink class device에는 공통 sysfs enable entry가 있으며 예제에서는 `20010000.etb/enable_sink`에 1을 써 ETB sink를 enable하고 읽어서 상태 1을 확인한다.

boot 시점의 etm3x driver는 첫 address comparator에 `_stext`와 `_etext`를 설정해 그 범위의 instruction을 trace한다. 따라서 source를 enable하면 즉시 trace capture가 시작된다.

예제는 `2201c000.ptm/enable_source`에 1을 쓴 뒤 ETB status를 읽는다. depth는 `0x2000`, control은 `0x1`이고 RAM write pointer `0x19d3`가 움직이는 것이 실제 data 수집의 표식이다.

trace 수집을 멈출 때는 같은 source `enable_source`에 0을 쓴다.

ETB buffer 내용은 `/dev/20010000.etb`에서 `dd`로 직접 읽어 `cstrace.bin`에 저장할 수 있다. 예제는 32768 byte를 복사한다. 이 file은 `ptm2human`, DS-5 또는 Trace32로 decompress할 수 있다.

DS-5 예제는 변수를 특정 값까지 증가시키는 loop의 instruction trace다. `PUSH`, stack 조정, 변수 load/store, compare, increment, 조건 branch가 여러 차례 반복되고 중간 timestamp가 기록된다. 끝에는 context 관련 register load와 function call, stack 복구, return과 마지막 timestamp가 보인다.

단순한 loop지만 각 instruction address, opcode, branch 실행 여부와 timestamp를 복원할 수 있어 CoreSight가 제공하는 분석 가능성을 보여 준다.

How to use the tracer modules
-----------------------------

There are two ways to use the Coresight framework:

1. using the perf cmd line tools.
2. interacting directly with the Coresight devices using the sysFS interface.

Preference is given to the former as using the sysFS interface
requires a deep understanding of the Coresight HW.  The following sections
provide details on using both methods.

Using the sysFS interface
~~~~~~~~~~~~~~~~~~~~~~~~~

Before trace collection can start, a coresight sink needs to be identified.
There is no limit on the amount of sinks (nor sources) that can be enabled at
any given moment.  As a generic operation, all device pertaining to the sink
class will have an "active" entry in sysfs::

    root:/sys/bus/coresight/devices# ls
    replicator  20030000.tpiu    2201c000.ptm  2203c000.etm  2203e000.etm
    20010000.etb         20040000.funnel  2201d000.ptm  2203d000.etm
    root:/sys/bus/coresight/devices# ls 20010000.etb
    enable_sink  status  trigger_cntr
    root:/sys/bus/coresight/devices# echo 1 > 20010000.etb/enable_sink
    root:/sys/bus/coresight/devices# cat 20010000.etb/enable_sink
    1
    root:/sys/bus/coresight/devices#

At boot time the current etm3x driver will configure the first address
comparator with "_stext" and "_etext", essentially tracing any instruction
that falls within that range.  As such "enabling" a source will immediately
trigger a trace capture::

    root:/sys/bus/coresight/devices# echo 1 > 2201c000.ptm/enable_source
    root:/sys/bus/coresight/devices# cat 2201c000.ptm/enable_source
    1
    root:/sys/bus/coresight/devices# cat 20010000.etb/status
    Depth:          0x2000
    Status:         0x1
    RAM read ptr:   0x0
    RAM wrt ptr:    0x19d3   <----- The write pointer is moving
    Trigger cnt:    0x0
    Control:        0x1
    Flush status:   0x0
    Flush ctrl:     0x2001
    root:/sys/bus/coresight/devices#

Trace collection is stopped the same way::

    root:/sys/bus/coresight/devices# echo 0 > 2201c000.ptm/enable_source
    root:/sys/bus/coresight/devices#

The content of the ETB buffer can be harvested directly from /dev::

    root:/sys/bus/coresight/devices# dd if=/dev/20010000.etb \
    of=~/cstrace.bin
    64+0 records in
    64+0 records out
    32768 bytes (33 kB) copied, 0.00125258 s, 26.2 MB/s
    root:/sys/bus/coresight/devices#

The file cstrace.bin can be decompressed using "ptm2human", DS-5 or Trace32.

Following is a DS-5 output of an experimental loop that increments a variable up
to a certain value.  The example is simple and yet provides a glimpse of the
wealth of possibilities that coresight provides.
::

    Info                                    Tracing enabled
    Instruction     106378866       0x8026B53C      E52DE004        false   PUSH     {lr}
    Instruction     0       0x8026B540      E24DD00C        false   SUB      sp,sp,#0xc
    Instruction     0       0x8026B544      E3A03000        false   MOV      r3,#0
    Instruction     0       0x8026B548      E58D3004        false   STR      r3,[sp,#4]
    Instruction     0       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
    Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
    Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
    Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
    Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
    Timestamp                                       Timestamp: 17106715833
    Instruction     319     0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
    Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
    Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
    Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
    Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
    Instruction     9       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
    Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
    Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
    Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
    Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
    Instruction     7       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
    Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
    Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
    Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
    Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
    Instruction     7       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
    Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
    Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
    Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
    Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
    Instruction     10      0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
    Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
    Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
    Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
    Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
    Instruction     6       0x8026B560      EE1D3F30        false   MRC      p15,#0x0,r3,c13,c0,#1
    Instruction     0       0x8026B564      E1A0100D        false   MOV      r1,sp
    Instruction     0       0x8026B568      E3C12D7F        false   BIC      r2,r1,#0x1fc0
    Instruction     0       0x8026B56C      E3C2203F        false   BIC      r2,r2,#0x3f
    Instruction     0       0x8026B570      E59D1004        false   LDR      r1,[sp,#4]
    Instruction     0       0x8026B574      E59F0010        false   LDR      r0,[pc,#16] ; [0x8026B58C] = 0x80550368
    Instruction     0       0x8026B578      E592200C        false   LDR      r2,[r2,#0xc]
    Instruction     0       0x8026B57C      E59221D0        false   LDR      r2,[r2,#0x1d0]
    Instruction     0       0x8026B580      EB07A4CF        true    BL       {pc}+0x1e9344 ; 0x804548c4
    Info                                    Tracing enabled
    Instruction     13570831        0x8026B584      E28DD00C        false   ADD      sp,sp,#0xc
    Instruction     0       0x8026B588      E8BD8000        true    LDM      sp!,{pc}
    Timestamp                                       Timestamp: 17107041535

Perf PMU abstraction과 sink 선택

CoreSight tracer는 Perf framework에서 Performance Monitoring Unit(PMU) abstraction으로 표현된다. Perf는 관심 process가 schedule되는 시점에 맞춰 tracing enable 시점을 제어한다.

system에 CoreSight PMU가 구성되면 `perf list pmu` 결과에 `cs_etm//` kernel PMU event가 나타난다. tracer 수가 보통 processor core 수만큼 있더라도 `cs_etm` PMU는 한 번만 표시된다.

CoreSight PMU는 다른 PMU와 같은 형식을 사용한다. PMU 이름 뒤 slash `/.../` 안에 configuration option을 넣는다.

Perf는 적절한 sink를 자동 선택하지만 보통 sink가 여러 개이므로 `@` prefix의 특별 configuration으로 이름을 지정할 수 있다.

사용 가능한 sink는 `$SYSFS/bus/event_source/devices/cs_etm/sinks/`에 나열된다. 예제는 `tmc_etf0`, `tmc_etr0`, `tpiu0`을 보여 주며 `perf record -e cs_etm/@tmc_etr0/u --per-thread program`으로 TMC-ETR0을 명시한다.

추가 Perf CoreSight 예제는 OpenCSD GitHub repository의 `HOWTO.md`를 참조한다.

Using perf framework
~~~~~~~~~~~~~~~~~~~~

Coresight tracers are represented using the Perf framework's Performance
Monitoring Unit (PMU) abstraction.  As such the perf framework takes charge of
controlling when tracing gets enabled based on when the process of interest is
scheduled.  When configured in a system, Coresight PMUs will be listed when
queried by the perf command line tool:

	linaro@linaro-nano:~$ ./perf list pmu

		List of pre-defined events (to be used in -e):

		cs_etm//                                    [Kernel PMU event]

Regardless of the number of tracers available in a system (usually equal to the
amount of processor cores), the "cs_etm" PMU will be listed only once.

A Coresight PMU works the same way as any other PMU, i.e the name of the PMU is
provided along with configuration options within forward slashes '/' (see
`Config option formats`_).

Advanced Perf framework usage
-----------------------------

Sink selection
~~~~~~~~~~~~~~

An appropriate sink will be selected automatically for use with Perf, but since
there will typically be more than one sink, the name of the sink to use may be
specified as a special config option prefixed with '@'.

The available sinks are listed in sysFS under
($SYSFS)/bus/event_source/devices/cs_etm/sinks/::

	root@localhost:/sys/bus/event_source/devices/cs_etm/sinks# ls
	tmc_etf0  tmc_etr0  tpiu0

	root@linaro-nano:~# perf record -e cs_etm/@tmc_etr0/u --per-thread program

More information on the above and other example on how to use Coresight with
the perf tools can be found in the "HOWTO.md" file of the openCSD gitHub
repository [#third]_.

Instruction 합성·PID trace와 AutoFDO

Perf는 program trace를 기록하고 분석할 수 있다. `perf record`에서 `cs_etm` event와 sink를 지정해 실행을 기록한다. `perf report`와 `perf script`는 instruction trace에서 instruction 및 branch event를 합성해 실행을 분석한다.

`perf inject`는 trace data를 합성 event로 바꿀 수 있고 `--itrace`가 event type과 frequency를 제어한다. 이 문서 시점에는 64-bit program만 지원하며 32-bit Arm instruction decode에는 추가 작업이 필요하다.

kernel은 PID를 PE ContextID register에 쓸 수 있다. EL1에서 실행하는 kernel은 `CONTEXTIDR_EL1`에 PID를 저장한다. Arm VHE를 구현해 kernel이 virtualization host로 EL2에서 실행하면 `CONTEXTIDR_EL2`에 저장한다.

Perf PMU format은 ETM이 이 값을 trace data에 넣도록 programming한다. `contextid1`은 EL1과 EL2 kernel 모두에서 사용할 수 있다. EL1에서는 kernel PID trace를 enable하고 EL2에서는 guest application PID를 trace한다.

`contextid2`는 kernel이 EL2에서 실행할 때만 쓸 수 있고 EL2 kernel PID tracing을 enable한다. `contextid`는 현재 kernel EL의 PID trace option alias로 EL1에서는 `contextid1`, EL2에서는 `contextid2`와 같다.

Perf는 관련 EL의 PID tracing을 위해 `contextid`를 자동 enable한다. EL2에서는 host와 guest PID를 모두 trace하려면 `contextid1`과 `contextid2`를 동시에 설정할 수 있으며 예제는 `perf record -e cs_etm/contextid1,contextid2/u -- vm`이다.

Feedback Directed Optimization용 coverage를 만들 때 `perf inject --itrace --strip`은 원 trace data를 제거하고 합성 event로 대체한다.

AutoFDO 예제는 `autofdo`와 GCC 5가 필요하다. 먼저 bubble sort를 `-O3`로 build해 CPU 2에서 실행하면 5910 ms다. CoreSight recording을 추가한 실행은 12543 ms이고 69.640 MB `perf.data`를 기록한다.

이어서 `perf inject --itrace=il64 --strip`으로 `inj.data`를 만들고 `create_gcov`로 `sort.gcov` profile을 생성한다. `-fauto-profile=sort.gcov`로 다시 build한 `sort_autofdo` 실행은 예제에서 5806 ms다.

AutoFDO analysis using the perf tools
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

perf can be used to record and analyze trace of programs.

Execution can be recorded using 'perf record' with the cs_etm event,
specifying the name of the sink to record to, e.g::

    perf record -e cs_etm//u --per-thread

The 'perf report' and 'perf script' commands can be used to analyze execution,
synthesizing instruction and branch events from the instruction trace.
'perf inject' can be used to replace the trace data with the synthesized events.
The --itrace option controls the type and frequency of synthesized events
(see perf documentation).

Note that only 64-bit programs are currently supported - further work is
required to support instruction decode of 32-bit Arm programs.

Tracing PID
~~~~~~~~~~~

The kernel can be built to write the PID value into the PE ContextID registers.
For a kernel running at EL1, the PID is stored in CONTEXTIDR_EL1.  A PE may
implement Arm Virtualization Host Extensions (VHE), which the kernel can
run at EL2 as a virtualisation host; in this case, the PID value is stored in
CONTEXTIDR_EL2.

perf provides PMU formats that program the ETM to insert these values into the
trace data; the PMU formats are defined as below:

  "contextid1": Available on both EL1 kernel and EL2 kernel.  When the
                kernel is running at EL1, "contextid1" enables the PID
                tracing; when the kernel is running at EL2, this enables
                tracing the PID of guest applications.

  "contextid2": Only usable when the kernel is running at EL2.  When
                selected, enables PID tracing on EL2 kernel.

  "contextid":  Will be an alias for the option that enables PID
                tracing.  I.e,
                contextid == contextid1, on EL1 kernel.
                contextid == contextid2, on EL2 kernel.

perf will always enable PID tracing at the relevant EL, this is accomplished by
automatically enable the "contextid" config - but for EL2 it is possible to make
specific adjustments using configs "contextid1" and "contextid2", E.g. if a user
wants to trace PIDs for both host and guest, the two configs "contextid1" and
"contextid2" can be set at the same time:

  perf record -e cs_etm/contextid1,contextid2/u -- vm


Generating coverage files for Feedback Directed Optimization: AutoFDO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

'perf inject' accepts the --itrace option in which case tracing data is
removed and replaced with the synthesized events. e.g.
::

	perf inject --itrace --strip -i perf.data -o perf.data.new

Below is an example of using ARM ETM for autoFDO.  It requires autofdo
(https://github.com/google/autofdo) and gcc version 5.  The bubble
sort example is from the AutoFDO tutorial (https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
::

	$ gcc-5 -O3 sort.c -o sort
	$ taskset -c 2 ./sort
	Bubble sorting array of 30000 elements
	5910 ms

	$ perf record -e cs_etm//u --per-thread taskset -c 2 ./sort
	Bubble sorting array of 30000 elements
	12543 ms
	[ perf record: Woken up 35 times to write data ]
	[ perf record: Captured and wrote 69.640 MB perf.data ]

	$ perf inject -i perf.data -o inj.data --itrace=il64 --strip
	$ create_gcov --binary=./sort --profile=inj.data --gcov=sort.gcov -gcov_version=1
	$ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
	$ taskset -c 2 ./sort_autofdo
	Bubble sorting array of 30000 elements
	5806 ms

cs_etm configuration option format

다음 option은 Perf command line의 `cs_etm/.../` slash 사이에 넣어 기능을 enable한다. 같은 목록은 `/sys/bus/event_source/devices/cs_etm/format/`에도 있다.

`branch_broadcast`는 system-wide `ETM_MODE_BB`의 session-local version이다. `cycacc`, `retstack`, `timestamp`도 각각 `ETMv4_MODE_CYCACC`, `ETM_MODE_RETURNSTACK`, `ETMv4_MODE_TIMESTAMP`의 session-local version이다.

`contextid`, `contextid1`, `contextid2`는 PID tracing option이다.

`configid`는 custom configuration을 선택하는 implementation detail이며 직접 쓰지 않는다. `preset`은 custom configuration parameter를 override한다. 두 option은 `coresight-config.rst`의 Perf configuration 사용법을 참조한다.

`sinkid`는 sink 선택 문자열의 hash이며 `@` notation을 쓰면 자동 설정된다. 내부 implementation detail이므로 직접 사용하지 않는다.

`cc_threshold`는 cycle count threshold다. 값이 없거나 0이면 기본 `0x100`을 사용한다. 입력값이 `TRCIDR3.CCITMIN`이 나타내는 최소 cycle threshold보다 작으면 최소값을 대신 사용한다.

Config option formats
~~~~~~~~~~~~~~~~~~~~~

The following strings can be provided between // on the perf command line to enable various options.
They are also listed in the folder /sys/bus/event_source/devices/cs_etm/format/

.. list-table::
   :header-rows: 1

   * - Option
     - Description
   * - branch_broadcast
     - Session local version of the system wide setting:
       :ref:`ETM_MODE_BB <coresight-branch-broadcast>`
   * - contextid
     - See `Tracing PID`_
   * - contextid1
     - See `Tracing PID`_
   * - contextid2
     - See `Tracing PID`_
   * - configid
     - Selection for a custom configuration. This is an implementation detail and not used directly,
       see :ref:`trace/coresight/coresight-config:Using Configurations in perf`
   * - preset
     - Override for parameters in a custom configuration, see
       :ref:`trace/coresight/coresight-config:Using Configurations in perf`
   * - sinkid
     - Hashed version of the string to select a sink, automatically set when using the @ notation.
       This is an internal implementation detail and is not used directly, see `Using perf
       framework`_.
   * - cycacc
     - Session local version of the system wide setting: :ref:`ETMv4_MODE_CYCACC
       <coresight-cycle-accurate>`
   * - retstack
     - Session local version of the system wide setting: :ref:`ETM_MODE_RETURNSTACK
       <coresight-return-stack>`
   * - timestamp
     - Session local version of the system wide setting: :ref:`ETMv4_MODE_TIMESTAMP
       <coresight-timestamp>`
   * - cc_threshold
     - Cycle count threshold value. If nothing is provided here or the provided value is 0, then the
       default value i.e 0x100 will be used. If provided value is less than minimum cycles threshold
       value, as indicated via TRCIDR3.CCITMIN, then the minimum value will be used instead.

STM·CTI/CTM과 system configuration manager

System Trace Macrocell(STM) module 사용법은 다른 tracer와 같다. 차이는 program control flow가 아니라 client가 trace capture를 구동한다는 점이다.

STM tracer의 세부 항목은 sysfs에서 확인하고 각 entry 설명은 `Documentation/ABI/testing/sysfs-bus-coresight-devices-stm`을 참조한다. 예제 `stm0` directory에는 `enable_source`, hardware event와 port select/enable, management, trace ID 등이 있다.

다른 source와 마찬가지로 STM 사용 전에 sink를 선택해 enable하고 STM source를 enable해야 한다. 예제는 `tmc_etf0/enable_sink`과 `stm0/enable_source`에 각각 1을 쓴다.

이후 user-space application은 generic STM API가 제공하는 devfs interface `/dev/stm0`에서 channel을 요청하고 사용할 수 있다. generic API 세부 사항은 `Documentation/trace/stm.rst`에 있다.

CTI는 개별 CTI와 component 사이에 trigger signal 집합을 제공하고 CTM channel을 통해 모든 CTI 사이에 이를 전파한다. 사용법은 `Documentation/trace/coresight/coresight-ect.rst`에 별도로 설명한다.

CoreSight component는 programming option이 많고 system 전체에서 서로 상호작용하도록 설정할 수 있는 복잡한 device다.

CoreSight System Configuration manager는 이러한 복잡한 programming configuration을 선택해 Perf와 sysfs에서 쉽게 사용하도록 한다. 자세한 내용은 `Documentation/trace/coresight/coresight-config.rst`를 참조한다.

문서 footnote는 STM sysfs ABI, generic STM API, Linaro perf-opencsd repository, CTI/CTM 문서, CoreSight configuration 문서의 source path와 URL을 제공한다.

How to use the STM module
-------------------------

Using the System Trace Macrocell module is the same as the tracers - the only
difference is that clients are driving the trace capture rather
than the program flow through the code.

As with any other CoreSight component, specifics about the STM tracer can be
found in sysfs with more information on each entry being found in [#first]_::

    root@genericarmv8:~# ls /sys/bus/coresight/devices/stm0
    enable_source   hwevent_select  port_enable     subsystem       uevent
    hwevent_enable  mgmt            port_select     traceid
    root@genericarmv8:~#

Like any other source a sink needs to be identified and the STM enabled before
being used::

    root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/tmc_etf0/enable_sink
    root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/stm0/enable_source

From there user space applications can request and use channels using the devfs
interface provided for that purpose by the generic STM API::

    root@genericarmv8:~# ls -l /dev/stm0
    crw-------    1 root     root       10,  61 Jan  3 18:11 /dev/stm0
    root@genericarmv8:~#

Details on how to use the generic STM API can be found here:
- Documentation/trace/stm.rst [#second]_.

The CTI & CTM Modules
---------------------

The CTI (Cross Trigger Interface) provides a set of trigger signals between
individual CTIs and components, and can propagate these between all CTIs via
channels on the CTM (Cross Trigger Matrix).

A separate documentation file is provided to explain the use of these devices.
(Documentation/trace/coresight/coresight-ect.rst) [#fourth]_.

CoreSight System Configuration
------------------------------

CoreSight components can be complex devices with many programming options.
Furthermore, components can be programmed to interact with each other across the
complete system.

A CoreSight System Configuration manager is provided to allow these complex programming
configurations to be selected and used easily from perf and sysfs.

See the separate document for further information.
(Documentation/trace/coresight/coresight-config.rst) [#fifth]_.


.. [#first] Documentation/ABI/testing/sysfs-bus-coresight-devices-stm

.. [#second] Documentation/trace/stm.rst

.. [#third] https://github.com/Linaro/perf-opencsd

.. [#fourth] Documentation/trace/coresight/coresight-ect.rst

.. [#fifth] Documentation/trace/coresight/coresight-config.rst