Also an exciting side note: Since I spoke at NWA Tech Fest, there was a major Podman release.

Before I lean into this, I should probably also note that I’m not associated with the Podman project. I’m just a hella-enthusiastic user.

The talk intro

Docker is not the only container tool, it isn’t even the default on a lot of modern Linux. My favorite alternative can run your containers with no root and no daemon anywhere on the box. I’m not, however, trying to sell anyone on switching. I deal with Docker daily for clients, my local user is in the docker group right on the machine where I’m composing this post now, and before any of this I ran FreeBSD jails. This is about knowing the alternative, because Docker gets all the attention and I think the other option deserves some as well.

I didn’t switch to Podman out of ideology. I’ve run Fedora and the uBlue atomic distros (Bluefin and Bazzite) for the last year or so, and they ship Podman by default. If you’re on Fedora, RHEL, or anything in that family, you may already have Podman installed and not know it. Maybe it’s worth a tinker?

A container is just a process

First myth to kill: a container is not a virtual machine. No guest kernel, no virtual hardware, no hypervisor. A container is just a regular Linux process running on your kernel. That’s it. The magic is entirely in what the kernel lets that process see and use.

There are basically two kernel features that do all the work:

• Namespaces control what the process can see: its own process list, its own network, its own filesystem mounts, its own users. The process thinks it’s alone on the machine.
• cgroups control what it can use: how much CPU, memory, and I/O. That’s what stops one container from eating all the resources.

Everything else is plumbing on top of those two. Because a container is just a process, it shares your host’s kernel, which is why containers start in milliseconds and weigh megabytes instead of gigabytes. It’s also why the kernel is your trust boundary.

An image (the thing you actually ship and pull) is filesystem layers plus metadata describing how to run them. You pull an image, you run a container from it.

This didn’t just appear in 2013. chroot showed up in Unix around 1979(ish?) and changed what a process saw as the root of the filesystem. BSD jails and Solaris Zones took it further in the early 2000s. The real Linux ingredients, namespaces and cgroups, landed in the kernel between 2006 and 2008. LXC built the first real toolset on them in 2008. Then Docker arrived in 2013 and made the whole thing usable, which won the mindshare. Credit where it’s due. In 2015 the Open Container Initiative standardized the image and runtime formats, keeping us from being locked into a single vendor. Podman itself showed up in 2018.

Meet Podman

The first big difference from Docker is the daemon, or rather the lack of one. Docker runs a central daemon, dockerd, as root, all the time, and your docker command just talks to it. Podman has no daemon by default. It’s a CLI binary that forks and execs the container directly as your user. No always-on root process in the middle. If Podman itself crashed, your running containers wouldn’t care, because Podman was never babysitting them. (Before the WELL ACKSHUALLY shows up: Podman can run a socket service for API compatibility when something expects the Docker socket, but that’s opt-in. The point I’m trying to make is that Docker requires a long-running root daemon and Podman doesn’t.)

Podman uses the same OCI images as Docker, the same registries (Docker Hub, Quay, GHCR), and the CLI is meant to be one-to-one. Most days you can literally alias docker=podman and your muscle memory and scripts keep working: podman run, podman ps, podman build, and so forth translate well.

Where that falls apart is Compose. docker-compose and compose files lean on Docker-specific behavior and the daemon socket, and the Podman side can get fuzzy on anything non-trivial. If you live in compose files, budget time for the rough edges.

Which leans into where Docker is genuinely smoother in my opinion: ubiquity. Every tutorial/Stack Overflow answer assumes Docker, and the ecosystem is enormous (projects like linuxserver.io have containerized basically everything with Docker as the assumption). A lot of Docker’s quirks exist because it was the early shaper of all this and we live in the world it built. The rough edges are historical…

One Podman feature with no Docker equivalent: pods. A pod is a group of containers that share a network namespace, so they talk over localhost and you manage them as a unit. That’s where the name comes from, Podman is the pod manager. A Podman pod is single-host. It borrows the Kubernetes pod idea but there’s no scheduling or any of that (and it’s not Docker Swarm either). Podman gives you the pod shape locally, and you can (usually) hand that straight to k8s later.

Rootless, the part that matters to me

This is the heart of it. Everything I’m about to describe runs as my normal user without sudo or root.

The part that was hardest to wrap my head around was the UIDs. Inside the container, the process can be root, UID 0, with all the package-manager and file-ownership powers it expects. But out on the host, that’s just me, an unprivileged user. The thing that reconciles those two facts is the user namespace, a namespace specifically for user IDs that remaps them between the container and the host.

When your account is set up, you get a range of subordinate UIDs in /etc/subuid. On my machine it’s cmhobbs:524288:65536, meaning I own 65,536 fake UIDs starting at 524288. Those are the IDs the container gets to play with. So container root (UID 0) maps to my real user (UID 1000), and container UIDs 1 and up map into that subuid range. Root in the container == me outside the container.

If something breaks out of a rootless container, it lands as my unprivileged user, not as host root. Compare that to the classic Docker setup, where a breakout from a root daemon can mean the whole machine. So the nice thing here is that the blast radius shrinks. Same containers, dramatically smaller worst case.

LIke most things in life, it isn’t free. The main thing you give up is that as a normal user you can’t bind ports below 1024, so a rootless container can’t just grab port 80 or 53. At home I took the blunt path and lowered a kernel parameter so my rootless Pi-hole could bind DNS:

net.ipv4.ip_unprivileged_port_start = 53

It works, but it’s hella janky becasue it lowers the bar for every unprivileged process on the box, not just mine. In production I’d do it properly, with one privileged door out front (like a reverse proxy) and every app rootless on a high port behind it. There are cleaner options too (systemd socket activation, firewall redirects), but “concentrate the privilege in one place” is what you want to do.

An example

I don’t do live demos at conferences, so I was a coward and copied/pasted these simple examples into my slides and transferred them here:

$ whoami
cmhobbs

$ podman run -d -p 8080:80 nginx
256356789fe4...

$ curl -s localhost:8080
<!DOCTYPE html> ... Welcome to nginx!

$ podman ps
... web  nginx  Up  0.0.0.0:8080->80/tcp

I’m cmhobbs, a regular user, and I never typed sudo. I started an nginx web server, curled it, and there it is in podman ps, looking exactly like Docker.

Digging deeper to show the UID weirdness:

$ podman top web huser user
HUSER   USER
1000    root

$ podman exec web id
uid=0(root) gid=0(root) groups=0(root)...

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     524288      65536

podman top with the huser and user columns puts the host user next to the container user, process by process: host UID 1000 (me) is running as root inside the container. One process, two identities. podman exec id proves it from the inside: UID 0, root, all the root groups. And uid_map supports this:
The first line, 0 1000 1, says container UID 0 maps to host UID 1000 (me). The second says container UIDs from 1 up map into my subuid range starting at 524288.

So there it is: I’m not root, the container thinks it is, and the kernel quietly reconciles the two.

Running real services

A container you started by hand is a toy. Real infrastructure has to survive a reboot and start in order so we need an init system (which on every box I touch means systemd). Full disclosure: I do not love systemd. I wish occasional minor inconveniences on Lennart Poettering… may his sleeves dampen when he washes his hands. However it’s everywhere, I lost that fight years ago, and it’s genuinelygood at keeping rootless containers alive. None of that is the point of this talk…

The modern way to do this is Quadlets: instead of writing a systemd unit with podman run stuffed into ExecStart, you write a short declarative .container file and Podman’s generator emits the real .service unit for you.

1[Container]
2Image=docker.io/library/caddy:latest
3PublishPort=8080:80
4Volume=%h/caddy/data:/data:Z
5AutoUpdate=registry
6
7[Install]
8WantedBy=default.target

Three rootless-specific things to notice: %h expands to my home directory, the capital Z on the volume relabels it for SELinux so the container can read its own data, and AutoUpdate=registry gives hands-off image updates on a timer. One more gotcha: by default your user services die when you log out. loginctl enable-linger fixes that, so your containers start at boot and keep running with nobody logged in.

I’ve written this up already, so I won’t re-derive it here. If you want the full how-to, see my two SmallOps posts: Part 1 covers hand-rolled systemd units from jinja2 templates (the old way, for historical reference), and Part 2 covers the Quadlet replacement. In my home home this pattern runs Pi-hole, Jellyfin with GPU transcoding, Home Assistant talking to a Zigbee dongle, Uptime Kuma, FreshRSS, and a dozen other things, all rootless under systemd. The same Quadlet-and-Ansible pattern runs in production for some of my clients in industrial settings.

The foot guns I found along the way

I bumped into two sharp edges and both were straight out of the features that make rootless work.

Networking: a container can’t reach its own host by LAN IP

Kernel networking needs root, so rootless networking is faked in userspace by a helper called pasta (from the passt project). A pasta process sits next to each rootless container and bridges its network to the host by reusing the host’s own IP, routes, and DNS. It became Podman’s default in 5.0, replacing the older slirp4netns. Faster and lighter but pointy-er: by design, a pasta container cannot reach its own host using the host’s external LAN IP. Outbound to the rest of the world is fine. The host talking to itself by its LAN address gets refused.

I learned that because I run Uptime Kuma, rootless, on a box I call ser3, and it monitors the other services on that same box. One morning the dashboard lit up red: every check ser3 ran against itself was failing, and yet every service it was complaining about was up and serving traffic.

The checks resolve ser3’s hostname to its LAN IP, 192.168.50.15. They’d worked for weeks. The only thing that changed was a routine restart of the container, and that restart was the first time it came up under the new pasta default. The old container had quietly kept its slirp4netns networking the whole time, right up until it was restarted. The diagnostic, from inside the container:

$ curl 192.168.50.15:8123              ECONNREFUSED   # ser3 -> itself     FAILS
$ curl 192.168.50.190:9090             200 OK         # ser3 -> ser5       works
$ curl 1.1.1.1:443                     200 OK         # ser3 -> internet   works
$ curl host.containers.internal:8123   200 OK         # via pasta's name   works

Read that and it’s clean: only the box talking to itself by its external IP fails. Not the firewall, not the service being down, those would fail differently. That last line is the tell: host.containers.internal is a name pasta hands you that does route back to the host, so pasta isn’t broken, it’s doing exactly what it says.

For a monitoring box whose whole job is reaching host-local services, the fix was host networking:

--network=host

The container shares the host’s network stack outright, so localhost and the LAN IP both work again and it binds its ports directly. Making that change made every self-check recover.

The lesson I’d actually pull from this is about timing. The default flipped during an upgrade weeks earlier and nothing broke and it broke when a restart finally put the container on the new default. A long-running service can be living on old behavior long after you “changed” something. So when you bump something that moves a default, restart the affected services on purpose and watch them right then, instead of getting ambushed by an unrelated reboot weeks later.

File ownership: you’re reading the wrong side

The second edge comes straight from the UID remapping. The container writes a file “as root,” and on the host it lands owned by some UID up in the 524288 range, not by you. The everyday version: images from linuxserver.io (and there are a lot of them) expect a user with PUID 1000 and chown their data to it. Under rootless that 1000 is inside the namespace, so a plain chown 1000 from the host sets the wrong owner and the container can’t read its own files. Fix it from inside the namespace instead:

podman unshare chown -R 1000:1000 <path>

Podman enters the user namespaceand sets the owner the container will actually see. If ownership looks scrambled under rootless, you’re almost always looking at it from the host side where the remapped IDs are meaningless. Step into the namespace and the numbers make sense again.

A few footnotes

• If you don’t want the command line, there’s Podman Desktop, free and open source. It’ll manage your Docker containers and your Podman containers in the same window, so they coexist and you don’t have to choose on day one.
• Podman speaks Kubernetes: podman kube generate turns a container or pod into Kubernetes YAML, and podman kube play runs Kubernetes YAML locally. A nice way to develop against Kubernetes without standing up a cluster.

What to take away

• A container is just a process the kernel tells stories to.
• Rootless means container-root is only ever you, so a breakout stays contained and the blast radius shrinks.
• You can run real services this way today, on hardware you already own.

If you take one action: tonight, podman run something. You very likely already have Podman installed. Just try it.

References

• Podman Documentation
• Podman - ArchWiki
• pasta / passt project
• rootless containers
• podman-systemd.unit(5) (Quadlets)
• Make systemd better for Podman with Quadlet
• SmallOps Part 1: jinja2 systemd units
• SmallOps Part 2: Quadlets