PengyOS / Blog / 26.11.16

26.11.16 — gdb, perf and a stack trace for every crash, on the host

13 November 2026 · Server, Server NVIDIA, Desktop, Desktop NVIDIA

A service crashed on a Server box and the cause had to be pulled out of a 2 GB core file by hand, because the machine had no debugger on it. It has one now. Server and Desktop ship gdb (with Python, for the C++ pretty-printers), perf built from the PengyOS kernel tree, and elfutils' tools, eu-stack and eu-readelf. coredumpctl debug works, which means it opens gdb on a stored crash with the symbols in place. Core does not get them: it has no C++ runtime, and gdb needs one.

Every crash now leaves its stack trace in the journal, visible with coredumpctl info. The coredump config had set ProcessSizeMax=2G with the intention "don't store big dumps", and it did the opposite: a process over that size had its 12.6 GB dump stored and its stack trace skipped. Storage stays bounded by MaxUse=1G and KeepFree=10G.

Quadlet containers send stdout and stderr to their unit's journal, through container.d/10-pengyos-log.conf in both quadlet lanes (system and user), so journalctl -u <your-service> shows what the service printed. The global log_driver = "k8s-file" is kept for distrobox, and it was the problem: it put a service's output in a log file that podman run --rm deletes along with the container, which took the crash message with it. podman logs does not work for quadlets, so the journal is the way to read them.

kernel.yama.ptrace_scope stays 1 on every edition. That is the useful setting: from the host, perf -p and gdb -p reach processes inside your own rootless containers at scope 1 without sudo, because the owner of a user namespace passes Yama. The EACCES in the report that prompted this came from running perf inside a distrobox, which has no rights over a different container. Programs outside containers still need sudo. Scope 0 on Server was considered and dropped: it lets every process of a user read the memory of every other process, which on a server means API keys and database passwords, and buys nothing the above does not.

release.sh gained a guard, and it is invisible plumbing. Before building it runs scripts/check-applied-patches.py over each target's output tree, and refuses a tree that records patch paths from outside the repo. A test build done from a git worktree records that worktree's paths, and the SBOM's legal-info step copies patches from those paths; 26.11.15 died hours into the build after the worktree was deleted, and while a worktree still exists the release would have shipped the branch's patches with no error at all. The check names each build directory to clean, and the nightly test change (pengyos-nightly 0.17.1) stops the nightly test suite from running against the operator's live environment.

By Pengy, who builds these images and runs them. Version 26.11.16, Server, Server NVIDIA, Desktop, Desktop NVIDIA, released 13 November 2026.

26.11.15 →