LCU14-101: Coresight Overview --------------------------------------------------- Speaker: Mathieu Poirier Date: September 15, 2014 --------------------------------------------------- Coresight is the name given to a set of IP blocks providing hardware assisted tracing for ARM based SoCs. This presentation will give an introduction to the technology, how it works and offer a glimpse of the capabilities it offers. More specifically we will go over the components that are part of the architecture and how they are used. Next will be presented the framework Linaro is working on in an effort to provide consolidation and standardization of interfaces to the coresight subsystem. We will conclude with a status of our current upstreaming efforts and how we see the coming months unfolding. --------------------------------------------------- ★ Resources ★ Zerista: http://lcu14.zerista.com/event/member/137703 Google Event: https://plus.google.com/u/0/events/cvb85kqv10dsc4k3e0hcvbr6i58 Presentation: http://www.slideshare.net/linaroorg/lcu14-101-coresight-overview Video: https://www.youtube.com/watch?v=NzKPd3FByxI&list=UUIVqQKxCyQLJS6xvSmfndLA Etherpad: http://pad.linaro.org/p/lcu14-101 --------------------------------------------------- ★ Event Details ★ Linaro Connect USA - #LCU14 September 15-19th, 2014 Hyatt Regency San Francisco Airport --------------------------------------------------- http://www.linaro.org http://connect.linaro.org

1. LCU14-101: Coresight Overview
Mathieu Poirier, LCU14
LCU14 BURLINGAME

2. About this Presentation
● What is CoreSight
● Why Use CoreSight?
● Coresight Components and HW tracing
● Presentation of the Coresight Framework
● Minimal steps to enable tracing
● Upstreaming Strategy and Current Status
● What is Coming Next
● Question and Comments

3. Coresight in One Image
Source: ARM Ltd.

4. What is CoreSight
● A set of IP blocks added to ARM SoCs at synthesis time
● IP blocks provide HW assisted system tracing using an out of band
bus (ATB), i.e non-intrusive debug capabilities
● Very useful for system profiling and debugging without costly
external tools
● CoreSight blocks are enabled/disabled at will
● Synchronisation of many CoreSight sources allow for debugging of
very complex scenarios.
**Also includes JTAG specification, not covered in this
presentation

5. Why Using CoreSight?
● Enhanced tracing capabilities over common solutions:
● Decoupling of CPU power state and tracing engine
● Tuning and configuration of scenarios simply impossible to achieve with a JTAG
● Synchronisation of traces from different source → multi-dimensional debugging
● Two different trace format for enhance granularity (STM vs ETM)
● Some debugging scenarios can’t afford to stop the target for
debugging when a condition occurs:
● Assembly chain in production
● A vehicle in motion
● Some Systems don’t have a JTAG port or the port is not
accessible
● Server with 256 processors
● A probe in an oil well, somewhere in the field

6. Enhanced Tracing Capabilities
● JTAGs require to have a connection with a core, making it difficult
to trace power management intensive scenarios.
● Enhance tuning and capabilities:
● Address range comparators
● Single address comparator
● Context ID and Virtual ID tracing
● Start and Stop address matching
● Security states ex, secure, non-secure or both
● External input for global system awareness
● Three-state sequencer and counters
● Tunables can be mixed to make very complex cases:
● Start tracing only if in “secure mode”, between addresses 0xABC and 0xDEX, when
the ID of the running process is 456.

7. Two Different Trace Format
● STM: transform SW writes into STPv2
● ETM/PTM: represent program flows using PFT
CPU
kernel
user
space
DSP
ETB
TPIU
STM
SW writes
PFTv1.1
STPv2

8. Coresight Components
● Coresight IP blocks are divided in 4 categories:
● Source, link, sink and “global” (not official term)
● Source:
● Generate trace data (PFTv1.x and STPv2)
● ETM (Embedded Trace Macrocell), PTM (Program Trace Macrocell)
● STM (System Trace Macrocell)
● Links:
● Replicators - duplicate trace data, usually in 2 streams
● Funnel - aggregate trace data, usually 8 to 1
● Global:
● CTI - Cross Trigger Interface, for communication between entities
** The above list is not exclusive, there are other components **

9. A Typical Coresight System
Source: ARM Ltd

10. ETM/PTM Trace Output
● Trace output from ETM/PTMs are compressed to reduce the
amount of space taken in the trace buffer (ETB).
● The format is called Program Flow Trace (PFT)
● Only waypoints are recorded:
● A waypoint is a point where instruction execution by the processor might involve a
change in the program flow
● Can be an instruction (ex. indirect branches) or an event (ex. all exceptions)
● Each waypoint means that instructions between the current
waypoint address and the previous one have been executed.
● Using waypoints and the original program image, the program
flow can be reconstructed.
● Atoms are used to indicate waypoint instruction execution.

11. ETM/PTM Trace Output Decoding
● Decoding trace output is not to be taken lightly.
● Program Linaro has experimented with:
● ptm2human [1]:
● Not bad for an experimental exercise
● List received packet, i.e, sync, atoms, timestamps
● Does not interpret the content of the packet, so no waypoint decoding
● completely free, thumbs up to “hwangcc23”
● ARM DS-5:
● Yields very detailed traces
● Program execution can be reconstructed perfectly - see [2] for more information
● Other trace decoding programs:
● Texas Instrument’s Code Composer Studio
● Lauderbach Trace32
● Trace decoding is currently a big problem.
● Linaro is weighing options on how to proceed in this area.

12. STM Trace Decoding
● STM produces a trace stream conformant to the MIPI’s STPv2.
● For more information see:
● http://mipi.org/specifications/debug
● Tools to decode STM traces:
● ARM’s DS-5
● ST-Microelectronic’s STM-Probe
● Liewenthal Electronic’s Fido box

13. Linaro CoreSight Framework
● Linaro Started working on CoreSight in March after LCA2014
● Started from the initial framework submitted by Pratik Patel [3]
● The framework provides support for:
● source: ETMv3.3 to ETMv3.5 along with PTMv1.0 and PTMv1.1
● link: 8 port funnel and non-configurable replicator
● sink: ETBv1.0, TPIU and TMC (Trace Memory Controller)
● Support for STM and CTI will be submitted when the base
framework is accepted upstream.
● With the framework it is possible to aggregate, configure and
collect trace data seamlessly on a platform.

14. Where to get the code
● Coresight branch on git.linaro.og:
● https://git.linaro.org/kernel/coresight.git/
● Always look at the “master” branch for the latest code
● The code is rebased on the latest release
● The Coresight framework and drivers are under drivers/coresight/
● Menuconfig options can be found under “Kernel Hacking/Coresight Tracing Support”
● Earlier submission and initial RFC are also present.
● Upstream submission if you want to see what we’ve done:
● V1: http://thread.gmane.org/gmane.linux.kernel/1714785
● V2: http://thread.gmane.org/gmane.linux.kernel/1734361
● V3: https://lkml.org/lkml/2014/8/7/439
● V4: https://lkml.org/lkml/2014/8/20/618

15. CoreSight Framework Highlights
● Provides an easy integration via DT for any platforms with generic
coreSight components
● Plenty of flexibility for addition of, new non-generic devices
● A front-end/back-end configuration for common and architecture specific parts are very
likely to happen.
● Device specific method to access DT configuration can also be done easily
● Access via debugfs of all ETM/PTM configuration registers
● Processor state (hibernation) not coupled with ETM/PTM
configuration
● Multiple configuration of source and sink is supported.
● Provides interface for reporting the status of various components
and the gathering of meta data.

16. CoreSight Framework
● When the framework boots coresight devices are discovered
● The individual drivers will then register each newly found device
with the framework:
coresight-tpiu 20030000.tpiu: TPIU initialized
coresight-etb10 20010000.etb: ETB initialized
coresight-funnel 20040000.funnel: FUNNEL initialized
coresight-replicator replicator: REPLICATOR initialized ← non-configurable replicator don’t have an address map
coresight-etm3x 2201c000.ptm: ETM initialized
coresight-etm3x 2201d000.ptm: ETM initialized
coresight-etm3x 2203c000.etm: ETM initialized
coresight-etm3x 2203d000.etm: ETM initialized
coresight-etm3x 2203e000.etm: ETM initialized

17. CoreSight Framework
● When the kernel has booted, coresight devices and attributes are
found under /sys/kerne/debug/coresight:
root@linaro-developer:~# cd /sys/kernel/debug/coresight/
root@linaro-developer:/sys/kernel/debug/coresight# ls
00000000.replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm
20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm
root@linaro-developer:/sys/kernel/debug/coresight# ls 20010000.etb/
enable status trigger_cntr
root@linaro-developer:/sys/kernel/debug/coresight# ls 2201c000.ptm/
addr_acctype cntr_idx enable nr_ctxid_cmp seq_32_event
addr_idx cntr_rld_event enable_event reset seq_curr_state
addr_range cntr_rld_val etmsr seq_12_event status
addr_single cntr_val fifofull_level seq_13_event sync_freq
addr_start ctxid_idx mode seq_21_event timestamp_event
addr_stop ctxid_mask nr_addr_cmp seq_23_event traceid
cntr_event ctxid_val nr_cntr seq_31_event trigger_event
root@linaro-developer:/sys/kernel/debug/coresight#

18. CoreSight Framework
● By default the first address comparator is configured with “_stext”
and “_etext” → good health check of coresight implementation
root@linaro-developer:/sys/kernel/debug/coresight# ls
00000000.replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm
20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm
root@linaro-developer:/sys/kernel/debug/coresight# ls 20010000.etb
enable status trigger_cntr
root@linaro-developer:/sys/kernel/debug/coresight# cat 2201c000.ptm/addr_range
0x80008280 0x805d34d8 ← Addresses match with _stext and _etext in “System.map”
root@linaro-developer:/sys/kernel/debug/coresight# ls 20010000.etb/
enable status trigger_cntr
root@linaro-developer:/sys/kernel/debug/coresight# echo 1 > 20010000.etb/enable ← Enable at least one source
root@linaro-developer:/sys/kernel/debug/coresight# echo 1 > 2201c000.ptm/enable ← System already configure, simply enable
tracing
root@linaro-developer:/sys/kernel/debug/coresight# cat 20010000.etb/status
Depth: 0x2000
Status: 0x1
RAM read ptr: 0x0
RAM wrt ptr: 0x1aeb ← RAM write pointer is incrementing, i.e, traces are being collected
Trigger cnt: 0x0
Control: 0x1
Flush status: 0x0
Flush ctrl: 0x2001
root@linaro-developer:/sys/kernel/debug/coresight#

19. Enabling a “Path” using the Framework
Source: ARM Ltd

20. CoreSight Framework
● When satisfied with trace collection, simply harvest the content of
the ETB buffer:
root@linaro-developer:/sys/kernel/debug/coresight# echo 0 > 2201c000.ptm/enable
root@linaro-developer:/sys/kernel/debug/coresight# ls /dev/
Display all 166 possibilities? (y or n)
.initramfs/ snd/ tty55
.udev/ stderr tty56
20010000.etb stdin tty57
audio stdout tty58
block/ tty tty59
bus/ tty0 tty6
char/ tty1 tty60
root@linaro-developer:/sys/kernel/debug/coresight# dd if=/dev/20010000.etb of=~/cstrace.bin
64+0 records in
64+0 records out
32768 bytes (33 kB) copied, 0.00233771 s, 14.0 MB/s
root@linaro-developer:/sys/kernel/debug/coresight#
root@linaro-developer:/sys/kernel/debug/coresight# ll ~/cstrace.bin
-rw-r--r-- 1 root root 32768 Aug 22 16:00 /root/cstrace.bin ← File with the trace data, in PFT format
root@linaro-developer:/sys/kernel/debug/coresight#

21. Upstreaming Stargegy and Status
● CodeAurora’s submission created a lot of talk on STM and CTI
implementation
● Linaro’s goal was to establish a foundation without being impaired
by STM and CTI debates
● Concentrating on basic tracing would concentrate efforts and
discussions on the framework implementation
● STM and CTI will be coming as soon as the framework gets
accepted upstream.
● At this time we have been through 4 submission, each round
yielding very fruitful comments and ideas

22. What is next
● Currently supported platforms:
● ARM Vexpress-TC2
● TI beagle/beagleXM
● Huawei’s D01 board (upstreaming currently under way)
● New platforms we are currently working on:
● Qualcomm APQ8074
● TI UEVM5432 (OMAP5)
● Support for STM/STM500
● Need to find a way to manage channel allocation between user and kernel space
● How to model external input in a flexible and generic way?
● Userspace tool and libraries…
● CTI (Cross Trigger Interface)
● John Hunter (formely?) at TI published a good driver on github
● codeAurora also has one, need to look at both

23. What is Next (Cont’d)
● Support for ARMv8
● The framework has been developed on ARMv7, need to see how it will work on V8
● Devices, like STM500 ant ETMv4, are tailored for V8
● STM500 may turn out to be like STM
● ETMv4 is a very different implementation than ETMv3.x
● How to support the tracing of application code?
● Integration with other kernel subsystem, i.e Perf and Ftrace
● Follow what other architectures are doing and not re-invent the
wheel:
● Intel’s PT architecture [6][7]
● MIPS ?

24. Challenges (Preview of the Next Session)
● How data is represented in the trace stream depends on how the
trace source has been configure (cycle accurate, timestamp)
● Unlike ftrace, trace streams can’t simply be exported off platform and decoded
● There has to be knowledge about the configuration and the image in memory
● Commonly called the “metadata” problem
● Identification of trace stream decoder that is open and free
● ptm2human is a very good start.
● etm-objdump[4]]5] also did a very promising decoder
● Both need to be improved (and maintained)
● How do we control traces from the kernel?
● ftrace has the capability to start and stop traces using a kernel API
● Do we need the same mechanism for coresight?
● STM channel allocation and management

25. Question and Comment
[1]. https://github.com/hwangcc23/ptm2human
[2]. https://wiki.linaro.org/WorklingGroups/Kernel/Coresight/traceDecodingWithDS5
[3]. http://lists.infradead.org/pipermail/linux-arm-kernel/2012-December/138646.html
[4]. http://lists.linaro.org/pipermail/linaro-dev/2012-November/014439.html
[5]. http://lists.linaro.org/pipermail/linaro-dev/2012-November/014476.html
[6]. http://thread.gmane.org/gmane.linux.kernel/1771197
[7]. https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing

26. More about Linaro Connect: connect.linaro.org
Linaro members: www.linaro.org/members
More about Linaro: www.linaro.org/about/