Podman on ZFS
Posted on Wed 03 January 2024 in linux
When running rootless podman on a ZFS mount, it defaults to the vfs storage driver.
And while this works, it causes a huge waste of disk space by duplicating files for every layer.
Using overlay solves this when running rootfull, but this is currently not supported on ZFS.
It should arrive with ZFS 2.2, but that will probably only be available in Debian 13 Trixie somewhere in 2025.
However, we should have the same functionality with the FUSE-variant fuse-overlay, with an additional performance penalty for the userspace roundtrip.
For example, the current Home Assistant image in vfs mode takes up:
$ podman unshare rm -rf .local/share/containers
$ podman system reset
[...]
$ time podman pull ghcr.io/home-assistant/home-assistant:2023.12.4
Trying to pull ghcr.io/home-assistant/home-assistant:2023.12.4...
Getting image source signatures
Copying blob 2651e446c136 done
[...]
Copying config 003e99ae19 done
Writing manifest to image destination
Storing signatures
003e99ae19b0c68a9cbb0c28e68cf843e12513287d1945c1b0b3d1e0eb246973
real 16m1.174s
user 0m58.612s
sys 1m2.829s
$ du -chs .local/share/containers
19G .local/share/containers
Switching to the overlay storage driver is a huge win, both in time and in storage space:
$ cat .config/containers/storage.conf
[storage]
driver="overlay"
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
#mountopt = "noacl" # see below
$ podman unshare rm -rf .local/share/containers
$ podman system reset
[...]
$ time podman pull ghcr.io/home-assistant/home-assistant:2023.12.4
Trying to pull ghcr.io/home-assistant/home-assistant:2023.12.4...
Getting image source signatures
Copying blob 2651e446c136 done
[...]
Copying config 003e99ae19 done
Writing manifest to image destination
Storing signatures
003e99ae19b0c68a9cbb0c28e68cf843e12513287d1945c1b0b3d1e0eb246973
real 1m41.778s
user 0m33.646s
sys 0m9.357s
$ du -chs .local/share/containers
2.4G .local/share/containers
ACL hickup
But this doesn't seem to work for all images.
The Grafana image fails with a rather unspecific Operation not supported:
$ podman run -it --rm docker.io/grafana/grafana-oss:10.0.10
Trying to pull docker.io/grafana/grafana-oss:10.0.10...
Getting image source signatures
Copying blob bf463c6d6fd9 done
[...]
Copying config e5b83aa02b done
Writing manifest to image destination
Storing signatures
{"msg":"exec container process `/run.sh`: Operation not supported","level":"error","time":"2024-01-03T08:12:03.063449Z"}
After some looking around, I found these two GitHub issue that seems very similar. The problem described there was that the underlying ZFS file system had ACLs disabled, and the container tried to use them. And indeed, I have:
# zfs get aclmode /tank/podman
NAME PROPERTY VALUE SOURCE
tank/podman aclmode discard default
Adding the noacl mount option to storage.conf seems to solve that:
$ cat .config/containers/storage.conf
[storage]
driver="overlay"
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "noacl"
System-wide
Next, I wanted to configure this system-wide so all users would benefit from this.
I thought it would be as simple as moving the config file from ~/.config/containers/storage.conf to /etc/containers/storage.conf, but that does not seem the case:
$ podman run -it --rm docker.io/grafana/grafana-oss:10.0.10
Error: 'overlay' is not supported over zfs, a mount_program is required: backing file system is unsupported for this graph driver
Podman complains about the missing mount_program option, even though it is configured.
I haven't figured out why yet... To be continued
Building images
I also had some issues when building container images using podman build.
They resulted in an "Operation Not Supported" error when creating a layer.
I haven't figured out what exactly went wrong, but using buildah directly does work...