This tutorial will try to demonstrate the power of performance profiling features within the linux kernel. We will use the networking subsystem as a use case for demonstrating the performance profiling features.
The focus will be on the perf framework which has emerged as the de facto standard for performance analysis on Linux. The tutorial will cover well documented as well as hidden gems of the perf infrastructure.
We will provide a short overview of perf components and how they work by design. Afterwards the well defined tracepoints in the kernel will be explored, specifically the few available in the networking subsystem. We will proceed to learn how to add custom probes. Often finding a good spot in the kernel to add a probe could be difficult; therefore we will try to demonstrate on how one can try to find a good location for new probes.
Modern CPUs are constantly adding more features to their performance monitoring counters, as an example: PMC units were imprecise, so vendors of x86 compatible CPUs improved that situation by adding features to improve precision, called PEBS by Intel and IBS by AMD and what additional data they provide.
In the outlook we explore which additional raw counters the CPUs provide and how it is possible to sample events from the memory controller and northbridge and how data is moved around CPU caches (by tracing cache coherency protocols). We will discuss how to write perf scripting to achieve this goal. Participants should be able to understand the PMC documentation provided by the processor vendors and apply them to perf.
We will conclude with infrastructure issues around perf, mainly how to deal with debugging information in heterogeneous environments mainly using different and ever changing kernel versions, mismatching debug information and how to deal with that. I hope for insights from the audience because this problem isn't yet solved by us.
The tutorial is intended to be interactive and we hope to gain insights from people attending the tutorial as it applies to their environments.
Bring a laptop or remote access to fancier equipment and lets have fun.