fix: allow envbuilder on sysbox container runtime

[janLo]

This fixes #50 by temporary bind-mounting all readonly mounts within the MagicDir to keep them out of the way for kaniko.

After kaniko finished it's build, the original mountpoints are restored at their original location.

[janLo]

It probably needs a lot of polishing as I'm not native to the go ecosystem. I'm sorry about that. But I can confirm that it allows to successfully use envbuilder on top of a sysbox runtime.

[janLo]

To give a bit more of an explanation here:

The base-image I'm using in the envbuilder-dockerfile (debian:bookworm) contains a symlink from /lib to /usr/lib/ and kaniko tries to create this. To make way for this symlink, kaniko tries to remove /lib in the image - but lib is there because sysbox mounts /lib/modules/<kernelversion> read-only in the container. Removing the ro mount-point fails, of course.

I first attempted to add all ro mount points to the ignore list of kaniko - but this means that /lib/modules/<kernelversion> is on the ignore-list and this does not match the removal of /lib - neither with exact match nor as a prefix. I also think that adding /lib to the ignore list would be somewhat invasive.

So my "fix" is now to bind-mount all ro mounts which are not in /proc, /sys or the MagicDir into the MagicDir and unmount them at their original locations. After kaniko finished, I bind-mount them back in their original place. I considered to "just unmount" them - but I thought, that there was a reason for them and I couldn't oversee the implications.

One question that remains is, if this scheme should also apply to the RW mounts - as this would then be more close to how a "normal" container works (runtime-mounts are "on top" of the image-state).

[janLo]

🖐️

[janLo]

🙌

[mafredri]

Neat trick, TIL you can unmount the source of a bind-mount. 😄

[mafredri]

Do we really want to continue in both of these cases? Wouldn't it be safer to error out?

[janLo]

Looking at it now I think you might be right. On the other hand, the error will catch you later anyway when kaniko tries to delete the containing directory to make room for the base image.

But I do think fail fast is better. Feel free to make any changes you feel suitable.

[janLo]

Neat trick, TIL you can unmount the source of a bind-mount. 😄

There is stuff you learn in 20 years of Linux that you wish you would have never be exposed to 🫣

What do you think about RW mounts? It would mean that we could mount a persistent home directory In an envbuilder container without having to meddle with the ignorelist.

[johnstcn]

Creating a separate branch for resolving conflicts: https://github.com/coder/envbuilder/compare/cj/janl/sysbox-issues

Validated using sysbox v0.6.4 in a Lima VM using envbuilder's own devcontainer:

docker run -it --rm \
    -v /tmp/envbuilder:/workspaces \
    -e GIT_URL=https://github.com/coder/envbuilder \
    -e INIT_SCRIPT=bash \
    envbuilder:janl-sysbox-issues # local image

Note: dockerd does need to be started manually in the absence of any init or systemd. Some separate fiddling may be necessary there.

[johnstcn]

Continuing in #183

[johnstcn]

Closing this out as #183 has been merged.

b8e2947

Thanks for the contribution @janLo !