~ / sysadmin / linux / systemtap

Reading a running kernel with SystemTap

SystemTap lets a running Linux kernel be inspected from a small script. It is the difference between guessing why a system is slow and watching the function that is actually busy.

What it does

A probe script names the events to watch, such as a system call entry or a function call, and the information to print when they happen. The script is compiled into a module, loaded, and removed again when the run finishes. The kernel is not replaced and the machine is not rebooted.

Install and check

On a distribution, install the tool together with the matching debug information for the running kernel. Without the debug symbols, only a limited set of probes is available.

apt install systemtap
stap -v -e 'probe begin { printf("ready\\n"); exit() }'

A first script

Count system calls by name for a few seconds to see what a program is really doing.

probe syscall.* {
  calls[execname()]++
}
probe timer.s(5) {
  foreach (p in calls- limit 10)
    printf("%s %d\\n", p, calls[p])
  exit()
}

Where it pays off

• Finding which call is sleeping when a service feels frozen.
• Seeing file activity behind a slow batch job.
• Confirming that a suspected hot path really is hot.

Staying safe

A probe script runs with kernel privilege. Keep the workload small, avoid writing loops that run unbounded, and always test on a machine you can reboot. A script that runs too long can slow the whole system down.

The tracing tool keeps its documentation upstream; see the resources page for a starting point.