Process Tracing Commands

Similar to Network Commands, this post is meant to be a single point of reference for all the random ways of interacting with and tracing processes.

Windows

Essentials

From the video "1st 3 Windows IR Commmands" by BHIS:

netstat -abno
wmic process where processid=<pid> get commandline
tasklist.exe /m /fi "pid eq <pid>"

Linux

Essentials

Go-to's for General Use

These are commands I tend to rely on from muscle memory.

BSD-style process listing:

ps aux

Process tree:

ps axjf

Securtiy info (AppArmor / SELinux):

ps axZ

Live process and resource usage:

# Not as fancy as htop but just as good (and seemingly always available)
top
# "m" cycles through memory usage dispaly options
# "u" will prompt you for a user to filter for
# "c" toggles full command line and paths
# "z" + "b" toggle color/mono and bold display options
# "s" lets you enter the refresh speed (e.g. 2.0, 0.1, all in seconds, 0.5 is a good speed)
# "M" sort by memory usage
# "N" sort by PID number
# "P" sort by CPU usage
# "T" sort by TIME+
# Use "R" to reverse the sort order

# More robust process and resource usage, likely needs installed via package manager
htop

bpftrace

Trace new processes via exec() syscalls. Uses bpftrace/eBPF

bpftrace is a way to interact with the Linux BPF subsystem through LLVM.

github.com/iovisor, redirects to github.com/bpftrace
bpftrace.org/tutorial-one-liners

Below are a number of *.bt commands. These commands are effectively scripts wrapping around bpftrace to monitor a specific thing (they literally start with #!/usr/bin/env bpftrace).

You can see everything available on the system with:

find /usr/sbin/ -type f -name "*.bt"

This doesn't include the *-bpfcc commands that are also under /usr/sbin/. Depending on the system, these may be called directly, or indirectly (pending review).

These are excellent debugging and threat hunting tools.

Installation

# https://github.com/bpftrace/bpftrace#quick-start
sudo apt install bpftrace
sudo dnf install bpftrace

# Nightly AppImage
declare -A suffixes=([x86_64]="X64" [amd64]="AMD64");
declare prefix="bpftrace/bpftrace/workflows/binary/master/bpftrace";
declare url="https://nightly.link/${prefix}-${suffixes[$(uname -m)]}.zip";
curl -L -o bpftrace.zip "${url}" && unzip bpftrace.zip

Practical Usage

Detect and trace execution events on a system.

# Trace new processes via exec() syscalls.
sudo execsnoop.bt

# This focuses more narrowly than `execsnoop.bt` by printing all `bash` commands system-wide.
sudo bashreadline.bt

Detect and trace filesystem activity.

# Trace all open(),openat(),openat2() syscalls. (Noisy!)
sudo opensnoop.bt
sudo opensnoop.bt | grep -F '/etc/some/path'

# Trace the stat() syscall, showing which processes are attempting to stat which files.
sudo statsnoop.bt

# Trace directory entry cache (dcache) lookups. (Noisy!)
sudo dcsnoop.bt

Detect and trace network activity.

# Trace TCP active connections (connect())
sudo tcpconnect.bt

# Trace TCP passive connections (accept())
sudo tcpaccept.bt

Expanded Usage

You'll notice the .bt scripts themselves are not too long to read, but may be complicated to understand and write. The language is fully documented here, which means we can start learning interactively with something like Claude in a Virtual Machine.

AI Usage

claude-sonnet-5 helped research and modify the code below.

When tracing connect() calls with tcpconnect.bt, I noticed the COMM did not always resolve to the "thing" that's creating it. For example, Firefox network calls would display as Socket Thread instead of firefox.

To patch /usr/sbin/tcpconnect.bt to change this behavior, at the very bottom of the script, add $task and $pname variables that will resolve the "comm" string to the "group_leader" string, which should be the "thing" that spawns it. This almost always is the plane name of the "thing", such as firefox, chrome, or wget. Finally just swap in $pname for comm in the print statement, and that's it.

It's recommended to just make a copy of the original file under /usr/local/bin/ for your modifications, so you keep your version separate.

    //SNIP

    // Add the two lines below, trace up to process name, not just the comm
    $task = (struct task_struct *)curtask;
    $pname = $task->group_leader->comm;

    time("%H:%M:%S ");
    printf("%-8d %-16s ", pid, $pname); /* replace comm with $pname */
    printf("%-39s %-6d %-39s %-6d\n", $saddr, $lport, $daddr, $dport);
  }
}

Optionally, add #include <linux/sched.h> to the include directives. The changes above may work fine without this, but it seems this may be necessary on some older systems (pending review).

#ifndef BPFTRACE_HAVE_BTF
#include <linux/socket.h>
#include <linux/sched.h> /* add this line */
#include <net/sock.h>

Two things that would make this even better, resolving IPs to hostnames and getting the process path.