POWER and Z (among others)

[tianon]

Hey folks -- we don't have a way to officially consume other architectures yet for official images (initial work for that sort of thing over in docker-library/official-images#2289 / docker-library/go-dockerlibrary#2), but there's some interest in at least getting some initial images up for testing, etc of other platforms (specifically from the IBM folks for POWER and Z, cc @duglin @estesp), especially including building Docker's own upstream packages for some of these other architectures.

Are there official published rootfs tarballs I could consume to build something for their needs in the interim, or is that something we could look to add in the future? What amount of work does that represent, and how can we (official images team) and/or IBM help out with that? Thanks! ❤️

cc @flavio @cyphar @jordimassaguerpla

[tboerger]

I have started to provide the rootfs months ago, you can see the rootfs for various architectures like these for tumbleweed at https://build.opensuse.org/project/show/Virtualization:containers:images:openSUSE-Tumbleweed

[tboerger]

I forgot to mention, the only missing part for your git based workflow are branches within this repository

[tianon]

Oh, nice! So, I think the release files I'm looking for would be the ones under https://build.opensuse.org/package/binaries/Virtualization:containers:images:openSUSE-42.2/openSUSE-42.2-docker-image?repository=images, right? (https://build.opensuse.org/package/binaries/Virtualization:containers:images:openSUSE-Tumbleweed/openSUSE-Tumbleweed-docker-image?repository=images for Tumbleweed, etc etc)

As I noted, official images don't have a way to consume these officially (yet), but I do want to wire up my temporary automation to be pulling these in for creating armhf/opensuse, s390x/opensuse, etc.

Is there an easy way to programatically determine the URLs I'll need to download without scraping that HTML page for the full ...-1.0.0-Build2.18.tar.xz bits?

[cyphar]

@tianon Note that we are currently undergoing a lot of work within KIWI to make it output proper Docker images (docker save-style) using umoci and skopeo. In particular look at OSInside/kiwi#216 which now means that base images are built using umoci and then converted to Docker images with skopeo. Eventually these are going to be exported by OBS as tar files, so the current Dockerfile system is going to be a bit different in the future.

As for your question about programmatically figuring out the URLs, you could go and parse the necessary repodata files (which are all understandable by rpm and by zypper if you wanted to do it that way). In addition, OBS exports other data you could try to parse as well. But I'm not sure this pull-based model (where you have to pull all of these different parts yourself) is going to work all that well for you, given that we're also working on some sort of publishing pipeline for container images.

What would be awesome is if there was some programmatic way for us to tell you that there is an update of an openSUSE image (pull requests on GitHub aren't a very good updating system).

[jordimassaguerpla]

Hi @tianon : There is always a file without the "Build" suffix which is actually a link to the latest Build, so you can use that one. You can have a job look for the date of that file and pull it when it changes.

[tianon]

@jordimassaguerpla ahhh, thanks -- http://download.opensuse.org/repositories/Virtualization:/containers:/images:/openSUSE-Tumbleweed/images/ is a much better link for seeing the actual artifacts 😄 👍 ❤️

See https://hub.docker.com/r/armhf/opensuse/, https://hub.docker.com/r/aarch64/opensuse/, and https://hub.docker.com/r/ppc64le/opensuse/. 👍 (s390x is WIP because my s390x host is a bit on the fritz)

@cyphar yeah, I agree that for base images, the PR method doesn't really work as well as for images that don't contain a rootfs, but we haven't figured out something better yet (and the update cadence can't be too high or the automated builds folks get overwhelmed with rebuilds) 😞

[clnperez]

Huge thanks @tianon for pushing those ppc64le images to dockerhub! 🌷

@tboerger It looks like maybe the repos in the images need to be tweaked. I'm unable to install curl, for example.

# zypper refresh
Repository 'NON-OSS' is up to date. 
Repository 'OSS' is up to date. 
All repositories have been refreshed.

bash-4.4# zypper search curl
Loading repository data...
Reading installed packages...

S | Name     | Summary                          | Type   
--+----------+----------------------------------+--------
i | libcurl4 | Version 4 of cURL shared library | package

# zypper repos 
Repository priorities are without effect. All enabled repositories share the same priority.

# | Alias   | Name    | Enabled | GPG Check | Refresh
--+---------+---------+---------+-----------+--------
1 | non-oss | NON-OSS | Yes     | (r ) Yes  | Yes    
2 | oss     | OSS     | Yes     | (r ) Yes  | Yes

Edit: This is an issue with tumbleweed and 42.1 (the only two I've tried).

[cyphar]

@clnperez Thomas probably isn't the best person to ping about issues with images, given that he's moved on from working for SUSE (he's at OwnCloud now). While he's still technically a maintainer, we have a containers team at SUSE which works on this stuff as our full-time jobs.

To answer your question, the package situation for s390x and ppc64le is quite complicated for openSUSE. Basically, in order to use them you need to enable certain "ports" repositories -- which should be enabled in our images. Can you verify what OSS actually points to (what is the output of zypper lr OSS)?

[tboerger]

Thomas probably isn't the best person to ping about issues with images, given that he's moved on from working for SUSE (he's at OwnCloud now). While he's still technically a maintainer, we have a containers team at SUSE which works on this stuff as our full-time jobs.

I think he just asked me because I directly responded to that :P

[clnperez]

@cyphar Here ya go:

~> docker run ppc64le/opensuse:42.1 zypper lr OSS
Alias          : oss                                                          
Name           : OSS                                                          
URI            : http://download.opensuse.org/distribution/leap/42.1/repo/oss/
Enabled        : Yes                                                          
GPG Check      : ( p) Yes                                                     
Priority       : 99                                                           
Auto-refresh   : On                                                           
Keep Packages  : Off                                                          
Type           : yast2                                                        
GPG Key URI    :                                                              
Path Prefix    : /                                                            
Parent Service :                                                              
Repo Info Path : /etc/zypp/repos.d/oss.repo                                   
MD Cache Path  : /var/cache/zypp/raw/oss  

It sounds like we need to just add something like http://download.opensuse.org/ports/ppc/distribution/13.2/repo/oss/suse/ppc64le/, but could you point me to the right place for 42.1? I'm more familiar with SLES than OpenSuSE so I don't know how the releases are setup.

[clnperez]

@cyphar I found this: http://download.opensuse.org/ports/ppc/distribution/leap/42.1/repo/oss/
and it works. 😃

[cyphar]

I was going to ask if you would do something like zypper ar obs://openSUSE:Leap:42.1:Ports obs-ports if that fixed anything. I'll ping @jordimassaguerpla and @davidcassany whether they can figure out why those repos aren't enabled by default.

[flavio]

Just a quick note, people from IBM approached me and I manually pushed an openSUSE tumbleweed image here.

The image was built with a simple Dockerfile like this one:

FROM scratch
ADD opensuse-s390x-root.tar.xz /

The repositories bundled inside of it are working fine, it looks like @jordimassaguerpla and @davidcassany fixed that in the meantime.

Once we are done with an internal milestone we should invest time into using umoci to build native docker base images. This will allow us to push the image from our current x86_64 infrastructure.

[clnperez]

@cyphar I tried the command you pasted to see if it would be helpful, and before I could get to it, I first tried installing curl (to exactly recreate). There were new repo issues with checksums on tumbleweed (not with 42.1) on ppc64le:
Warning: Digest verification failed for file 'appdata-icons.tar.gz'
Warning: Digest verification failed for file 'appdata-screenshots.tar'
etc.

But in the end that command didn't add the right repo:

bash-4.2# zypper ar obs://openSUSE:Leap:42.1:Ports obs-ports
Guessed: platform = openSUSE_$releasever
Adding repository 'obs-ports' .......................................................................................................................[done]
Repository 'obs-ports' successfully added
Enabled     : Yes                                                                                
Autorefresh : No                                                                                 
GPG Check   : Yes                                                                                
URI         : http://download.opensuse.org/repositories/openSUSE:/Leap:/42.1:/Ports/openSUSE_42.1

bash-4.2# zypper install curl
Error building the cache:
[obs-ports|http://download.opensuse.org/repositories/openSUSE:/Leap:/42.1:/Ports/openSUSE_42.1] Valid metadata not found at specified URL
Warning: Skipping repository 'obs-ports' because of the above error.
Loading repository data...
Reading installed packages...
'curl' not found in package names. Trying capabilities.
No provider of 'curl' found.
Resolving package dependencies...

Nothing to do.

Let me know if you want me to try anything else.

[CrimsonGlory]

Would be nice to have mongodb image on hub.docker.com/u/ppc64le

[clnperez]

Thanks for the feedback @CrimsonGlory

[yosifkit]

We're finally officializing a lot of the multiarch work we've been doing, and as such we think we've finally got some decent examples that are worth looking at for how openSUSE might be enabled for multiarch 🙂 We also have a multiarch section in the official-images README.md.

docker-library/official-images#3024 (ubuntu) and docker-library/official-images#3031 (debian) are our first two "base" images with multiarch enabled (no manifest lists yet, just official multiarch specification so we can build/push from a single source of truth -- manifest lists are on the TODO shortly though); @tianon set up https://github.com/tianon/docker-brew-alpine-multiarch as another example of how this could go for Alpine, using the minirootfs tarballs that have been officially released (with https://github.com/tianon/docker-brew-alpine-multiarch/blob/master/alpine being an example library file for it).

We are happy to help with any of this! 🎉

[flavio]

This is great, we are definitely going to join the party :)

[jordimassaguerpla]

A PR for fixing the repos in tumbleweed:

openSUSE/docker-containers#66

Regarding the official multiarch section, I understand we could have different directories or branches in our github repo for the different archs, right? Since openSUSE is a "From scratch" image.

[tianon]

Yeah, different branches or different directories are fine -- whatever works best for your flow!

[tianon]

For Debian, Ubuntu, and BusyBox, I went with different branches since it made it easier to work with, but directories are fine too! (as seen in the example I made in https://github.com/tianon/docker-brew-alpine-multiarch for how Alpine could work)

[tianon]

Anything we can do to help here? 😄 (just a friendly prod 👍 ❤️)

[flavio]

@davidcassany is currently working in converting our internal build pipeline to use a new native docker build feature he added to KIWI. Adding the new architectures will follow along.

[utzb]

Hello @flavio and @davidcassany , we just realized that https://download.opensuse.org/ports/zsystems/ contains tumbleweed packages, but no repository for 42.2 packages on s390x. Is that just a glitch and a matter of activating, or do you expect it will not happen until the next release?

[flavio]

Hello @flavio and @davidcassany , we just realized that https://download.opensuse.org/ports/zsystems/ contains tumbleweed packages, but no repository for 42.2 packages on s390x. Is that just a glitch and a matter of activating, or do you expect it will not happen until the next release?

Last time we checked OBS had s390x repositories only for Tumbleweed. I don't know if the situation changed in the meantime, we will double check while working on this issue.

[davidcassany]

@tianon, as @flavio said I am working on updating the pipeline we have to produce our Docker images; also, as @cyphar mentioned here months ago, our current workflow is changing to better integrate the Open Build Service and KIWI (using umoci and skopeo) to produce native Docker images (native in the sense that the resulting tarball is ready to be loaded docker load -i myimage.tar.xz). So I am wondering if do we have any chance to be capable to direclty submit builded images into docker library?

[tianon]

See docker-library/official-images#527 for an older (still relevant) discussion of some issues with us doing that.

We'd also love to support Git-LFS, but as discussed in https://github.com/docker-library/official-images/issues/1095, it didn't support git archive last we tested (which might have changed since I last tested, but I'm not very hopeful).

I have been considering allowing for a rootfs to be hosted outside Git and simply including in the library/xxx file a reference to where to get it and a checksum of some kind for verifying it (with pre-baked Dockerfile contents, or something), but even that won't help with the docker load-able tarballs you're generating.

Up until now, I've been extracting the rootfs from the released tarballs (https://github.com/docker-library/oi-janky-groovy/blob/6e1191f17080ca600e59a1cdbfd8a0b289e4f8a1/multiarch/target-opensuse-pipeline.groovy#L74-L99) to get preview images up, which is where the current ARCH/opensuse images are coming from (as noted in this thread). I'd be happy to take over the automation of pushing rootfs tarballs into Git based on the published tarballs too, if that's something which would be of use or help.

[davidcassany]

@tianon thanks the response and the references, I read the thread you suggested (docker-library/official-images#527) and I understand there are some reasons to prevent the suggested there strategy. I was wondering if what I have done here and here is more in line of what you would expect.

I realize this is quite similar of what is suggest in the mentioned thread, but I believe this still could be a valid option if some constraints are included in the tests you perform, for instance our images do no include a multilayered rootfs, so IMHO there is no need for any further special analysis of what you are already doing. What you think?

[tianon]

I mean, FROM davidcassany/amd64@sha256:aef1d1c1151d14fd09143812b6d65bb5bfbee626f4c88d1a6e87b224c11ccc67 is a clever way to solve the tarballs-in-Git problem, but it doesn't solve anything about the review problem (and actually makes the review more complicated, since we then have to inspect the images themselves to ensure the metadata and layers are sane), and will then require additional exemptions in https://github.com/docker-library/official-images#repeatability.

Multi-stage builds are something we've talked about using to help with this issue, although we aren't using a new enough version of Docker on the official build servers yet to implement them fully. Here's an all-out example of what that approach might look like with Ubuntu's official rootfs tarballs (https://partner-images.canonical.com/core/):

FROM debian:stretch-slim AS fetch

RUN set -ex; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
		ca-certificates \
		gnupg2 dirmngr \
		wget \
	; \
	rm -rf /var/lib/apt/lists/*

ENV UBUNTU_FINGERPRINT D2EB44626FDDC30B513D5BB71A5D6C4C7DB87C81

RUN gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$UBUNTU_FINGERPRINT"

ENV UBUNTU_SUITE xenial
ENV UBUNTU_SERIAL 20170802

RUN set -eux; \
	\
	wget "https://partner-images.canonical.com/core/${UBUNTU_SUITE}/${UBUNTU_SERIAL}/SHA256SUMS"; \
	wget "https://partner-images.canonical.com/core/${UBUNTU_SUITE}/${UBUNTU_SERIAL}/SHA256SUMS.gpg"; \
	gpg --batch --verify SHA256SUMS.gpg SHA256SUMS; \
	\
	arch="$(dpkg --print-architecture)"; \
	tarball="ubuntu-${UBUNTU_SUITE}-core-cloudimg-${arch}-root.tar.gz"; \
	wget -O "$tarball" "https://partner-images.canonical.com/core/${UBUNTU_SUITE}/${UBUNTU_SERIAL}/$tarball"; \
	grep -q " *$tarball\$" SHA256SUMS; \
	grep " *$tarball\$" SHA256SUMS | sha256sum -c -; \
	\
	mkdir /rootfs; \
	tar -xf "$tarball" -C /rootfs

# TODO add /usr/sbin/policy-rc.d
# TODO add /sbin/initctl
# TODO add /etc/dpkg/dpkg.cfg.d/...
# TODO add /etc/apt/apt.conf.d/...
# TODO purge /var/lib/apt/lists/*
# TODO update sources.list
# TODO update /run/systemd/container
# see https://github.com/tianon/docker-brew-ubuntu-core/blob/ea95f62628559b4dc13b179cce87f96b83d44238/update.sh#L57-L103
# (https://github.com/tianon/docker-brew-ubuntu-core/blob/8984e91c47abd923cf214fb7232b044106b39337/xenial/Dockerfile)

FROM scratch
COPY --from=fetch /rootfs /
CMD ["bash"]

[flavio]

Unfortunately this multi approach solution would not work because the tarball we are producing is now something like docker save -o foo.tar instead of docker export -o foo.tar. 😞

[cyphar]

@tianon The problem is that we are no longer producing simple tarballs. We use umoci to create an OCI image which contains all of the metadata and then convert it to a Docker image with skopeo. The result is that the final image is a full-on docker save image -- which is why we'd need to use FROM to include it in the image.

The thing is, the old method of just including a .tar file wasn't particularly good for reproduciblity either. What would be great is if we could point you at a docker save tar file, and then you can import it, but because the current official-library revolves around Dockerfiles this gets quite a bit complicated. Would it be possible to rectify the latter issue perhaps?

[vielmetti]

Putting my name in here to express an interest in seeing arm64 OpenSUSE images, as well as to query re status changes since early September.

[mnhauke]

I just tried arm32v7/opensuse and aarch64/opensuse and found some issues:

01 - Docker recently did some reorganization on their registry for 64bit ARM systems

See the deprecation notice in https://hub.docker.com/r/aarch64/opensuse/

The aarch64 organization is deprecated in favor of the more-specific arm64v8 organization, as per https://github.com/docker-library/official-images#architectures-other-than-amd64.

02 - arm32v7/opensuse:tumbleweed is broken

RPMs for the wrong platform (from the armv6hl repositores) are installed per default
http://download.opensuse.org/ports/armv7hl/tumbleweed/repo/oss/
should be used

03 - When arm32v7 docker images runs on aarch64 systems zypper cannot determine the correct architecture.

A workaround could be to force the architecture in zypp.conf
sed -i 's|# arch = s390|arch = armv7hl|g' /etc/zypp/zypp.conf
but there's probably a more elegant solution.

04 - Incorrect repositories for all architectures that are maintained in 'ports' are used.

Here are the correct ones for arm32v7 and aarch64:

armv7hl:

Leap 42.2

http://download.opensuse.org/ports/armv7hl/distribution/leap/42.2/repo/oss/ http://download.opensuse.org/ports/update/42.2/oss/openSUSE:Leap:42.2:Update.repo

Leap 42.3

http://download.opensuse.org/ports/armv7hl/distribution/leap/42.3/repo/oss/ http://download.opensuse.org/ports/update/42.3/oss/openSUSE:Leap:42.3:Update.repo

Tumbleweed

http://download.opensuse.org/ports/armv7hl/tumbleweed/repo/oss/ http://download.opensuse.org/ports/armv7hl/update/tumbleweed/openSUSE:Factory:Update.repo

aarch64:

Leap 42.2

http://download.opensuse.org/ports/aarch64/distribution/leap/42.2/repo/oss/ http://download.opensuse.org/ports/update/42.2/oss/openSUSE:Leap:42.2:Update.repo

Leap 42.3

http://download.opensuse.org/ports/aarch64/distribution/leap/42.3/repo/oss/ http://download.opensuse.org/ports/update/42.3/oss/openSUSE:Leap:42.3:Update.repo

Tumbleweed

http://download.opensuse.org/ports/aarch64/tumbleweed/repo/oss/ http://download.opensuse.org/ports/aarch64/update/tumbleweed/openSUSE:Factory:Update.repo

[cyphar]

@mnhauke In general we would recommend using images from the opensuse/ namespace -- we have automated jobs that push to it immediately after they're built by OBS. Unfortunately it doesn't look like we have opensuse/arm64 (or whatever appropriate ARM would be).

/cc @jordimassaguerpla @davidcassany

[michelmno]

FYI, I verified that two last openSUSE images for ppc64le are able to be pulled from https://store.docker.com/images/opensuse but still have inside bad zypper repos, so add question in previous reference openSUSE/docker-containers#66

[flavio]

@davidcassany can you take a look at that please?

[davidcassany]

@flavio this was solved last week here openSUSE/docker-containers#79 and closed the other mentioned issue, I just didn't realize this issue was still open.