A service crashed on a Server box and the cause had to be pulled out of a 2 GB core file by hand,
because the machine had no debugger on it. It has one now. Server and Desktop ship gdb (with Python,
for the C++ pretty-printers), perf built from the PengyOS kernel tree, and elfutils' tools, eu-stack
and eu-readelf. coredumpctl debug works, which means it opens gdb on a stored crash with the
symbols in place. Core does not get them: it has no C++ runtime, and gdb needs one.
Every crash now leaves its stack trace in the journal, visible with coredumpctl info. The coredump
config had set ProcessSizeMax=2G with the intention "don't store big dumps", and it did the
opposite: a process over that size had its 12.6 GB dump stored and its stack trace skipped. Storage
stays bounded by MaxUse=1G and KeepFree=10G.
Quadlet containers send stdout and stderr to their unit's journal, through
container.d/10-pengyos-log.conf in both quadlet lanes (system and user), so
journalctl -u <your-service> shows what the service printed. The global log_driver = "k8s-file"
is kept for distrobox, and it was the problem: it put a service's output in a log file that
podman run --rm deletes along with the container, which took the crash message with it. podman
logs does not work for quadlets, so the journal is the way to read them.
kernel.yama.ptrace_scope stays 1 on every edition. That is the useful setting: from the host,
perf -p and gdb -p reach processes inside your own rootless containers at scope 1 without sudo,
because the owner of a user namespace passes Yama. The EACCES in the report that prompted this came
from running perf inside a distrobox, which has no rights over a different container. Programs
outside containers still need sudo. Scope 0 on Server was considered and dropped: it lets every
process of a user read the memory of every other process, which on a server means API keys and
database passwords, and buys nothing the above does not.
release.sh gained a guard, and it is invisible plumbing. Before building it runs
scripts/check-applied-patches.py over each target's output tree, and refuses a tree that records
patch paths from outside the repo. A test build done from a git worktree records that worktree's
paths, and the SBOM's legal-info step copies patches from those paths; 26.11.15 died hours into the
build after the worktree was deleted, and while a worktree still exists the release would have
shipped the branch's patches with no error at all. The check names each build directory to clean,
and the nightly test change (pengyos-nightly 0.17.1) stops the nightly test suite from running
against the operator's live environment.