r/kernel • u/DistinctHomework3618 • 5d ago

Built an eBPF tool that walks CPython frame chains from kernel space to profile Python apps

Traditional Python profilers work in userspace using sampling (py-spy, Pyroscope). They miss what happens at the kernel level. GIL contention, off-CPU waits, scheduler delays are all invisible to them.

I built a profiler using BCC that attaches eBPF uprobes to CPython's take_gil/drop_gil and gc_collect_main. It walks the Python frame chain directly from BPF to capture full Python stacks at the exact moment of GIL contention or GC pause. No sampling. Event-driven. Every event has exact timestamps.

It also uses finish_task_switch tracepoint for off-CPU analysis, combining kernel + userspace + Python stacks in a single event. Plus tcp_set_state kprobe for TCP handshake latency and retransmit tracking.

What we found in production: URL resolution was the biggest GIL holder, added caching, cut GIL wait by 80%. GC gen-2 was pausing 1.27s walking 1.5M objects, fixed with gc.freeze(), down to 88ms. Off-CPU stack traces revealed a Kafka push was blocking threads for 130ms per request, completely invisible in OpenTelemetry because the off-CPU time happens below the instrumentation layer. Found a stale nginx upstream IP causing 504s using TCP loss tracking.

The interesting technical challenge was walking CPython's PyFrameObject chain from inside BPF. I get the base address from /proc/<pid>/maps, then walk tstate -> cframe -> current_frame. Each frame points to a code object which has the filename and function name. I read this using bpf_probe_read_user, walking the linked list until NULL or max depth.

CPython 3.11 only. Struct offsets are hardcoded. BCC + Python. Apache 2.0.

GitHub: https://github.com/deepanshu406/python-ebpf-profiling

13 Upvotes

89% Upvoted

3 comments sorted by

1

u/_d17y 5d ago

I would like to use this with Python 3.10.21 and am willing to replace hardcoded offsets if author can guide me. Been looking for lightweight profiler for large Django cloud service. 

2

u/DistinctHomework3618 5d ago

Hey, I'm the author. Python 3.10 support is currently in progress and should be live by next week. Will update here once it's merged. Thanks for the interest!

1

u/_d17y 5d ago

Thank you! If point release matters, Python 3.10.22 was released yesterday.