[incident] Alpine 3.20 seccomp issues (arm32v6, arm32v7, i386, riscv64, ppc64le)

[tianon]

Many of the builds for the recent Alpine 3.20 based images are currently failing due to a variant of #16829 (although interestingly mostly on less popular architectures such as 32bit x86/arm, riscv64, and ppc64le). We are aware of the issues, and are working on a fix. 👍

[tianon]

Got all my ducks in a row for builds of a newer Docker engine across all the architectures we support and the problem persists. I then recall that BuildKit needs the profile updates too, which might be why we're still seeing it. 🙃 🤞

https://github.com/moby/buildkit/blob/v0.12.5/go.mod#L31 is some semi-random commit between v24 and v25, which does not have any of moby/moby@9e10605 (v26 original merge), moby/moby@7a075ca (v25 backport), moby/moby@f11b55f (v24 backport), or moby/moby@5cc6748 (v23 backport) 😭

The good news is that https://github.com/moby/buildkit/blob/v0.13.2/go.mod#L32 is new enough (v25.0.3 is the version that received the backport). The bad news is that I refactored the way I'm building tianon/buildkit and it's temporarily lost a few architectures as a result, so I need to sort that out. 😭

However, the build I'm still seeing fail is using the default BuildKit-built-into-dockerd (just raw docker buildx build, no --builder), so it's also very, very possible (and maybe even likely?) that this is something I also need to chase but is unrelated to still seeing the build fail. 😭

[tianon]

Oh man, we probably need libseccomp2 version 2.5.5+ also, which is actually newer than Debian Stable/bookworm currently has.

[tianon]

seccomp/libseccomp@v2.5.4...v2.5.5#diff-b37807d701e0c96a05fa13d461c3f4ca7ca00e64b68358d7e5905d94b60858c2R75

Confirmed 😭

[tianon]

Further confirmed -- started an infosiftr/moby:i386 container on my host, tried docker build 'https://github.com/tianon/docker-bash.git#967275700d273c985448c889a4e825008cd1fb97:5.2' to reproduce the failure, then installed libseccomp2 version 2.5.5 from unstable and tried the build again with great success (so at least I can apply a workaround in my Docker-in-Docker images! still not sure about the BuildKit bits but one bridge at a time).

I also filed a bug against src:libseccomp in Debian asking for 2.5.5 (or the syscall updates) in either bookworm or bookworm-backports: https://bugs.debian.org/1071822

[tianon]

Ok, in theory, fixes are deployed everywhere now. I'll check back in next week and hopefully everything will be resolved and all these failing Alpine 3.20 builds will be green. 😄

Edit: I guess assuming I'm wrong about BuildKit 0.12 above 🙈

[tianon]

(Reader: he couldn't resist and he wasn't wrong about BuildKit above, so that's next week's project)

[LaurentGoderre]

Still seems to be failing on armv6 and v7

[tianon]

Doing some hacks in my personal build system to get more architectures on tianon/buildkit again (notably riscv64): tianon/bashbrew-tianon-meta@371fd85#diff-dec510d864246250be6cc8d33d2f9040f88ee65dfa31ee3b0e98b4e4ea1fe291

The other half I'll have to solve is using the host architecture for determining which BuildKit image to use instead of using the target architecture (which is a disparity we've had for a while that mostly doesn't matter but in this case becomes important since I'm not hacking it with fake entries anymore).

Edit: for the curious, that's accomplished via the following sadness (which I eventually hope to move to something that's committed but for now this is it):

# TODO implement proper cross-architecture support instead of this stupid hack 😭
jq '
	.[].arches[] |=
		if any(.tags[]; startswith("tianon/buildkit:")) then
			.froms |= map(select(startswith("golang:") | not))
			| .parents |= with_entries(select(.key | startswith("golang:") | not))
		else . end
' sources.json > sources-hacked.json
( set -x && diff -u sources.json sources-hacked.json ) || :
mv sources-hacked.json sources.json

[tianon]

The other half I'll have to solve is using the host architecture for determining which BuildKit image to use instead of using the target architecture (which is a disparity we've had for a while that mostly doesn't matter but in this case becomes important since I'm not hacking it with fake entries anymore).

Apparently, I've (fortuitously!) forgotten that I didn't actually implement this to that level, and our arm32v6 and arm32v5 builders are already running the arm32v7 image, so we're actually good to go with just that hack! 🎉

[tianon]

In theory, #16856 will finish fixing this (holding off closing this until actual verification though). 🤞

[yosifkit]

ppc64le was successful! And arm32v7. 🎉

[tianon]

This looks like it's settled enough to be closeable. 👍

[tianon]

I also filed a bug against src:libseccomp in Debian asking for 2.5.5 (or the syscall updates) in either bookworm or bookworm-backports: https://bugs.debian.org/1071822

This weekend's Debian stable update included this gem:

libseccomp (2.5.4-1+deb12u1) bookworm; urgency=medium

  * Add support for syscalls up to Linux 6.7. (Closes: #1071822)

 -- Felix Geyer <[email protected]>  Sat, 25 May 2024 20:36:52 +0200

🥳 ❤️