Kernel Upgrade 5.18.19 -> 5.19.4 broke monitoring host
Note: This is a Fedora 36 host, i haven't tested with my remaining Fedora 35 host yet.
Only a single container could be successfully started on that host, containers started afterwards would fail with:
Unmounting /var/lib/containers/storage/overlay/93e24cebe04e168477966613e96974874731374f18fce16543a5c2fb71b967d6/merged: invalid argument
Full debug log
INFO[0000] podman filtering at log level debug DEBU[0000] Called start.PersistentPreRunE(podman start gatus --log-level=debug) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is being used DEBU[0000] Cached value indicated that native-diff is not being used INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/crun" INFO[0000] Setting parallel job count to 4 DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is being used DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true DEBU[0000] Cached value indicated that idmapped mounts for overlay are supported DEBU[0000] Cached value indicated that idmapped mounts for overlay are supported DEBU[0000] Cached value indicated that volatile is being used ERRO[0000] Unmounting /var/lib/containers/storage/overlay/93e24cebe04e168477966613e96974874731374f18fce16543a5c2fb71b967d6/merged: invalid argument DEBU[0000] Failed to mount container "13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b": create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/93e24cebe04e168477966613e96974874731374f18fce16543a5c2fb71b967d6/mapped/0": invalid argument DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] Cleaning up container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] Container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b storage is already unmounted, skipping... DEBU[0000] Removing container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b DEBU[0000] Cleaning up container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] Container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b storage is already unmounted, skipping... DEBU[0000] Removing all exec sessions for container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b DEBU[0000] Container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b storage is already unmounted, skipping... Error: unable to start container "13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b": error mounting storage for container 13df7df36aac890000562fc845ab76b293427a05df898c0d8fbf103500516a0b: create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/93e24cebe04e168477966613e96974874731374f18fce16543a5c2fb71b967d6/mapped/0": invalid argument
podman system reset
didn't help either.
It seems like something has changed regarding overlay idmapped mounts. Needs further investigating.
Current fix: Rollback to 5.18.19 Kernel and possibly podman system reset
.
Edited by histalek