Rootless Containers with Podman
I gave a talk at NWA Tech Fest called “Rootless Containers with Podman.” This is the version that I congealed from the slides and my notes for the people who asked for it afterwards (and like a month later because I’m forgetful). The tone is probably a little too expository because I chopped stuff directly out of my speaking script with minor edits, so try to read it as if I’m yammering on at you at lunch and you can’t get away because your phone battery is flat. As an aside, I submitted a larger version of this talk to TXLF and it got accepted! I’ll be presenting on Day 2. Hope to see you there!
Also an exciting side note: Since I spoke at NWA Tech Fest, there was a major Podman release.
Before I lean into this, I should probably also note that I’m not associated with the Podman project. I’m just a hella-enthusiastic user.
The talk intro
Docker is not the only container tool,
it isn’t even the default on a lot of modern Linux. My favorite alternative can
run your containers with no root and no daemon anywhere on the box. I’m not,
however, trying to sell anyone on switching. I deal with Docker daily for
clients, my local user is in the docker group right on the machine where I’m
composing this post now, and before any of this I ran FreeBSD jails. This is
about knowing the alternative, because Docker gets all the attention and I think
the other option deserves some as well.
I didn’t switch to Podman out of ideology. I’ve run Fedora and the uBlue atomic distros (Bluefin and Bazzite) for the last year or so, and they ship Podman by default. If you’re on Fedora, RHEL, or anything in that family, you may already have Podman installed and not know it. Maybe it’s worth a tinker?
A container is just a process
First myth to kill: a container is not a virtual machine. No guest kernel, no virtual hardware, no hypervisor. A container is just a regular Linux process running on your kernel. That’s it. The magic is entirely in what the kernel lets that process see and use.
There are basically two kernel features that do all the work:
- Namespaces control what the process can see: its own process list, its own network, its own filesystem mounts, its own users. The process thinks it’s alone on the machine.
- cgroups control what it can use: how much CPU, memory, and I/O. That’s what stops one container from eating all the resources.
Everything else is plumbing on top of those two. Because a container is just a process, it shares your host’s kernel, which is why containers start in milliseconds and weigh megabytes instead of gigabytes. It’s also why the kernel is your trust boundary.
An image (the thing you actually ship and pull) is filesystem layers plus metadata describing how to run them. You pull an image, you run a container from it.
This didn’t just appear in 2013. chroot showed up in Unix around 1979(ish?)
and changed what a process saw as the root of the filesystem. BSD jails and
Solaris Zones took it further in the early 2000s. The real Linux ingredients,
namespaces and cgroups, landed in the kernel between 2006 and 2008. LXC built
the first real toolset on them in 2008. Then Docker arrived in 2013 and made
the whole thing usable, which won the mindshare. Credit where it’s due. In
2015 the Open Container Initiative standardized the image and runtime formats,
keeping us from being locked into a single vendor. Podman itself showed up in
2018.
Meet Podman
The first big difference from Docker is the daemon, or rather the lack of one.
Docker runs a central daemon, dockerd, as root, all the time, and your docker
command just talks to it. Podman has no daemon by default. It’s a CLI binary
that forks and execs the container directly as your user. No always-on root
process in the middle. If Podman itself crashed, your running containers wouldn’t
care, because Podman was never babysitting them. (Before the WELL ACKSHUALLY
shows up: Podman can run a socket service for API compatibility when something
expects the Docker socket, but that’s opt-in. The point I’m trying to make is
that Docker requires a long-running root daemon and Podman doesn’t.)
Podman uses the same OCI images as Docker, the same registries (Docker Hub,
Quay, GHCR), and the CLI is meant to be one-to-one. Most days you can literally
alias docker=podman and your muscle memory and scripts keep working: podman run, podman ps, podman build, and so forth translate well.
Where that falls apart is Compose. docker-compose and compose files lean on
Docker-specific behavior and the daemon socket, and the Podman side can get
fuzzy on anything non-trivial. If you live in compose files, budget time for
the rough edges.
Which leans into where Docker is genuinely smoother in my opinion: ubiquity. Every tutorial/Stack Overflow answer assumes Docker, and the ecosystem is enormous (projects like linuxserver.io have containerized basically everything with Docker as the assumption). A lot of Docker’s quirks exist because it was the early shaper of all this and we live in the world it built. The rough edges are historical…
One Podman feature with no Docker equivalent: pods. A pod is a group of containers that share a network namespace, so they talk over localhost and you manage them as a unit. That’s where the name comes from, Podman is the pod manager. A Podman pod is single-host. It borrows the Kubernetes pod idea but there’s no scheduling or any of that (and it’s not Docker Swarm either). Podman gives you the pod shape locally, and you can (usually) hand that straight to k8s later.
Rootless, the part that matters to me
This is the heart of it. Everything I’m about to describe runs as my normal user
without sudo or root.
The part that was hardest to wrap my head around was the UIDs. Inside the container, the process can be root, UID 0, with all the package-manager and file-ownership powers it expects. But out on the host, that’s just me, an unprivileged user. The thing that reconciles those two facts is the user namespace, a namespace specifically for user IDs that remaps them between the container and the host.
When your account is set up, you get a range of subordinate UIDs in
/etc/subuid. On my machine it’s cmhobbs:524288:65536, meaning I own 65,536
fake UIDs starting at 524288. Those are the IDs the container gets to play with.
So container root (UID 0) maps to my real user (UID 1000), and container UIDs 1
and up map into that subuid range. Root in the container == me outside the
container.
If something breaks out of a rootless container, it lands as my unprivileged user, not as host root. Compare that to the classic Docker setup, where a breakout from a root daemon can mean the whole machine. So the nice thing here is that the blast radius shrinks. Same containers, dramatically smaller worst case.
LIke most things in life, it isn’t free. The main thing you give up is that as a normal user you can’t bind ports below 1024, so a rootless container can’t just grab port 80 or 53. At home I took the blunt path and lowered a kernel parameter so my rootless Pi-hole could bind DNS:
net.ipv4.ip_unprivileged_port_start = 53It works, but it’s hella janky becasue it lowers the bar for every unprivileged process on the box, not just mine. In production I’d do it properly, with one privileged door out front (like a reverse proxy) and every app rootless on a high port behind it. There are cleaner options too (systemd socket activation, firewall redirects), but “concentrate the privilege in one place” is what you want to do.
An example
I don’t do live demos at conferences, so I was a coward and copied/pasted these simple examples into my slides and transferred them here:
$ whoami
cmhobbs
$ podman run -d -p 8080:80 nginx
256356789fe4...
$ curl -s localhost:8080
<!DOCTYPE html> ... Welcome to nginx!
$ podman ps
... web nginx Up 0.0.0.0:8080->80/tcpI’m cmhobbs, a regular user, and I never typed sudo. I started an nginx web
server, curled it, and there it is in podman ps, looking exactly like Docker.
Digging deeper to show the UID weirdness:
$ podman top web huser user
HUSER USER
1000 root
$ podman exec web id
uid=0(root) gid=0(root) groups=0(root)...
$ podman unshare cat /proc/self/uid_map
0 1000 1
1 524288 65536podman top with the huser and user columns puts the host user next to the
container user, process by process: host UID 1000 (me) is running as root
inside the container. One process, two identities. podman exec id proves it
from the inside: UID 0, root, all the root groups. And uid_map supports this:
The first line, 0 1000 1, says container UID 0 maps
to host UID 1000 (me). The second says container UIDs from 1 up map into my
subuid range starting at 524288.
So there it is: I’m not root, the container thinks it is, and the kernel quietly reconciles the two.
Running real services
A container you started by hand is a toy. Real infrastructure has to survive a reboot and start in order so we need an init system (which on every box I touch means systemd). Full disclosure: I do not love systemd. I wish occasional minor inconveniences on Lennart Poettering… may his sleeves dampen when he washes his hands. However it’s everywhere, I lost that fight years ago, and it’s genuinelygood at keeping rootless containers alive. None of that is the point of this talk…
The modern way to do this is Quadlets: instead of writing a systemd unit
with podman run stuffed into ExecStart, you write a short declarative
.container file and Podman’s generator emits the real .service unit for you.
1[Container]
2Image=docker.io/library/caddy:latest
3PublishPort=8080:80
4Volume=%h/caddy/data:/data:Z
5AutoUpdate=registry
6
7[Install]
8WantedBy=default.targetThree rootless-specific things to notice: %h expands to my home directory,
the capital Z on the volume relabels it for SELinux so the container can read
its own data, and AutoUpdate=registry gives hands-off image updates on a
timer. One more gotcha: by default your user services die when
you log out. loginctl enable-linger fixes that, so your containers start at
boot and keep running with nobody logged in.
I’ve written this up already, so I won’t re-derive it here. If you want the full how-to, see my two SmallOps posts: Part 1 covers hand-rolled systemd units from jinja2 templates (the old way, for historical reference), and Part 2 covers the Quadlet replacement. In my home home this pattern runs Pi-hole, Jellyfin with GPU transcoding, Home Assistant talking to a Zigbee dongle, Uptime Kuma, FreshRSS, and a dozen other things, all rootless under systemd. The same Quadlet-and-Ansible pattern runs in production for some of my clients in industrial settings.
The foot guns I found along the way
I bumped into two sharp edges and both were straight out of the features that make rootless work.
Networking: a container can’t reach its own host by LAN IP
Kernel networking needs root, so rootless networking is faked in userspace by a helper called pasta (from the passt project). A pasta process sits next to each rootless container and bridges its network to the host by reusing the host’s own IP, routes, and DNS. It became Podman’s default in 5.0, replacing the older slirp4netns. Faster and lighter but pointy-er: by design, a pasta container cannot reach its own host using the host’s external LAN IP. Outbound to the rest of the world is fine. The host talking to itself by its LAN address gets refused.
I learned that because I run Uptime Kuma, rootless, on a box I call ser3, and it monitors the other services on that same box. One morning the dashboard lit up red: every check ser3 ran against itself was failing, and yet every service it was complaining about was up and serving traffic.
The checks resolve ser3’s hostname to its LAN IP, 192.168.50.15. They’d
worked for weeks. The only thing that changed was a routine restart of the
container, and that restart was the first time it came up under the new pasta
default. The old container had quietly kept its slirp4netns networking the
whole time, right up until it was restarted. The diagnostic, from inside the
container:
$ curl 192.168.50.15:8123 ECONNREFUSED # ser3 -> itself FAILS
$ curl 192.168.50.190:9090 200 OK # ser3 -> ser5 works
$ curl 1.1.1.1:443 200 OK # ser3 -> internet works
$ curl host.containers.internal:8123 200 OK # via pasta's name worksRead that and it’s clean: only the box talking to itself by its external IP
fails. Not the firewall, not the service being down, those would
fail differently. That last line is the tell: host.containers.internal
is a name pasta hands you that does route back to the host, so pasta isn’t
broken, it’s doing exactly what it says.
For a monitoring box whose whole job is reaching host-local services, the fix was host networking:
--network=hostThe container shares the host’s network stack outright, so localhost and the LAN IP both work again and it binds its ports directly. Making that change made every self-check recover.
The lesson I’d actually pull from this is about timing. The default flipped during an upgrade weeks earlier and nothing broke and it broke when a restart finally put the container on the new default. A long-running service can be living on old behavior long after you “changed” something. So when you bump something that moves a default, restart the affected services on purpose and watch them right then, instead of getting ambushed by an unrelated reboot weeks later.
File ownership: you’re reading the wrong side
The second edge comes straight from the UID remapping. The container writes a
file “as root,” and on the host it lands owned by some UID up in the 524288
range, not by you. The everyday version: images from linuxserver.io (and there
are a lot of them) expect a user with PUID 1000 and chown their data to it.
Under rootless that 1000 is inside the namespace, so a plain chown 1000
from the host sets the wrong owner and the container can’t read its own files.
Fix it from inside the namespace instead:
podman unshare chown -R 1000:1000 <path>Podman enters the user namespaceand sets the owner the container will actually see. If ownership looks scrambled under rootless, you’re almost always looking at it from the host side where the remapped IDs are meaningless. Step into the namespace and the numbers make sense again.
A few footnotes
- If you don’t want the command line, there’s Podman Desktop, free and open source. It’ll manage your Docker containers and your Podman containers in the same window, so they coexist and you don’t have to choose on day one.
- Podman speaks Kubernetes:
podman kube generateturns a container or pod into Kubernetes YAML, andpodman kube playruns Kubernetes YAML locally. A nice way to develop against Kubernetes without standing up a cluster.
What to take away
- A container is just a process the kernel tells stories to.
- Rootless means container-root is only ever you, so a breakout stays contained and the blast radius shrinks.
- You can run real services this way today, on hardware you already own.
If you take one action: tonight, podman run something. You very likely
already have Podman installed. Just try it.
References
#podman #containers #rootless #linux #sysadmin #devops #security #smallops