[proposal] How to implement image storage for sealer using Skopeo

[jsparter]

Issue Description

background:

Now we always pull application images online at build stage, even if there are images in docker-daemon locally. And we pull blobs of images and manage all files by hands which is so inelegant. As #1874 said，we need to load offline images at build stage.

Skopeo is a image transporter, it can transport series of types of image from one to another, and skopeo copy ability can help us to implement load offline images. Skopeo operates on the following image and repository types: containers-storage:docker-reference、dir:path、docker://docker-reference、docker-archive:path[:docker-reference]、docker-daemon:docker-reference、oci:path:tag. Skopeo can transport images between any two types above:

skopeo copy containers-storage:docker.io/library/alpine:3.13 docker://sea.hub:5000/library/alpine:3.13
skopeo copy docker-daemon:ubuntu:rolling docker://sea.hub:5000/library/ubuntu:rolling

docker://docker-reference、 docker-daemon:docker-reference and oci:path:tag these three image types may match our needs. But, each image type mentioned above has different structure, docker://docker-reference needs there is a registry service running on.

So, there are three solutions tu implement image store i figured:

1. Running a registry in build stage, and use skopeo copy ability to transport images to registry. It can help us fully make use of Skopeo, and completely offload loading image capability to Skopeo. Obviously, this approach relies on the Docker and registry containers, and brings complexity to the build stage.
2. Partly make use of skopeo copy to connect docker-daemon and read local images, then still manage all files by hands. It's hard to implement, and still inelegant. In the long run, if the Registry store format changes, our maintenance costs will be high.
3. Load images to a middle type at build stage using skopeo copy ability, and load middle type image to registry at run stage affter registry is running using skopy copy directly. For example, use oci:path:tag as middle type, so there are two step as follow：

   skopeo copy docker-daemon:docker.io/library/alpine:3.13 oci:$rootfs/ociimages:library/alpine:3.13
   skopeo copy oci:$rootfs/ociimages:library/alpine:3.13 docker://sea.hub:5000/library/alpine:3.13
   

   In this way, we seperate registry and image store, if user want to use their own registry to store image (eg. harbor) in the future, we retain the ability to do that. Additional costs is that we move images twice, making run stage be slower.

Each solution has its inherent weaknesses, and the design needs to determine which one to use, or if there is a better one

Describe what feature you want

Additional context

[jsparter]

@starnop Is this had any discussion or conclution?

[jsparter]

[proposal] unify image storage

Background

At prasent, sealer pull and sealer build both have ability to pull and store images, but they have different storage backend and use different SDK, cause code redundance and structure sepration.

Describe what feature you want

Use buildah SDK to unify sealer pull and sealer build storage ability, while sealer pull store images in localhost, and sealer build store images in rootfs.
Cause buildah store images with OCI format, if users use CRI in addition to docker, downloaded images can be read directly by CRI from rootfs without registry. But distribute images with rootfs brings much more costs, so we always need a registry responsible for it.

In my opinion, best choise by now is like solution3 mentioned above which is loading and storing images with OCI format in rootfs (by build command) or localhost (by pull command), then move images to registry in running stage.

[starnop]

@starnop Is this had any discussion or conclution?

Sorry for the late reply. I will give comments today.

[starnop]

Nice Job!

I'll go over a few little details with you offline.
However, as one of the biggest concerns, in terms of compatibility, how do you think about it?

[starnop]

@jsparter Any update?

[jsparter]

@jsparter Any update?

Comparing Artifacthub, I noticed that project-zot/zot might be more suitable for our scenario, I will write a little comparison summary soon

[jsparter]

Update:
project-zot/zot can fit our need, and the following works successfully:

# run zot registry
docker run  -d -p 5000:5000 -v $(pwd)/registry:/var/lib/registry ghcr.io/project-zot/zot-linux-amd64:v1.4.3
# copy any images to $(pwd)/registry using skopeo
skopeo copy docker-daemon:busybox:latest oci:$(pwd)/registry/busybox:v1
# pull image from zot registry
docker pull localhost:5000/busybox:v1
# check if the pull was successful
docker images | grep busybox

[jsparter]

Discussion on the feasibility of sealer using oci registry

1. Comparison of basic capabilities of zot and docker registry

	docker registry	zot
HomePage	https://docs.docker.com/registry/	https://zotregistry.io/v1.4.3/general/concepts/
Authentication	√	√
multiple os/arch	√	√
image signatures	√	√
Storage	filesystem、azure、gcs、s3、swift、oss	filesystem、s3 （!any client that is compliant with the Distribution Specification can read from or write to a zot registry）
deploy	container、swarm、docker-compose	binary、container、helm

1.1 Performance comparison between zot and docker registry

To be finished

1.2 Storage format difference (eg: alpine:3.13、busybox:latest )

docker registry:

[root@k8s-master-32 v2]# tree
.
├── blobs
│   └── sha256
│       ├── 00
│       │   ├── 0048118155842e4c91f0498dd298b8e93dc3aecc7052d9882b76f48e311a76ba
│       │   │   └── data
······
└── repositories
    ├── alpine
    │   ├── _layers
    │   │   └── sha256
    │   │       ├── 6b5c5e00213a401b500630fd8a03f6964f033ef0e3a6845c83e780fcd5deda5c
    │   │       │   └── link
    │   │       └── 986be8ca46800cd6b9c0e9b71f9e8a32b200a6e56d719e731536fc43a2ab5b7e
    │   │           └── link
    │   ├── _manifests
    │   │   ├── revisions
    │   │   │   └── sha256
    │   │   │       └── 71f16231793003a03c18b2fb0a6b4b84ce23a1ce1b21f867f4b096218c0f0672
    │   │   │           └── link
    │   │   └── tags
    │   │       └── 3.13
    │   │           ├── current
    │   │           │   └── link
    │   │           └── index
    │   │               └── sha256
    │   │                   └── 71f16231793003a03c18b2fb0a6b4b84ce23a1ce1b21f867f4b096218c0f0672
    │   │                       └── link
    │   └── _uploads
    ├── busybox
    │   ├── _layers
    │   │   └── sha256
    │   │       ├── 076588f6d6ab00f95c93be0211468a144e61ad67bb63969c682c9cd24b8c4656
    │   │       │   └── link
    │   │       └── 827365c7baf137228e94bcfc6c47938b4ffde26c68c32bf3d3a7762cd04056a5
    │   │           └── link
    │   ├── _manifests
    │   │   ├── revisions
    │   │   │   └── sha256
    │   │   │       └── 634f3e56874d2226ded74b6488db32c3e879e91581e9c396ceabf39b7b282345
    │   │   │           └── link
    │   │   └── tags
    │   │       └── latest
    │   │           ├── current
    │   │           │   └── link
    │   │           └── index
    │   │               └── sha256
    │   │                   └── 634f3e56874d2226ded74b6488db32c3e879e91581e9c396ceabf39b7b282345
    │   │                       └── link
    │   └── _uploads

zot:

.
├── alpine
│   ├── blobs
│   │   └── sha256
│   │       ├── 05e07062ad2b305da1c3d3d8dda2388d6f4df948595dd85218c77007129eb9c3
│   │       ├── 986be8ca46800cd6b9c0e9b71f9e8a32b200a6e56d719e731536fc43a2ab5b7e
│   │       └── e0d2486fb1ef4e79e678c4cc950ac939d46fac4e4c38de5f467d29ba33a6b19e
│   ├── index.json
│   └── oci-layout
├── busybox
│   ├── blobs
│   │   └── sha256
│   │       ├── 076588f6d6ab00f95c93be0211468a144e61ad67bb63969c682c9cd24b8c4656
│   │       ├── 614a7d8579dab1b10b1d1b13fa15eef6929494dbba17284458057de9865f9486
│   │       └── bf86d23d3e5c04d7232d4ae49d11af8dec39854c7bbcc83fe2571f8730a9fea8
│   ├── index.json
│   └── oci-layout
└── cache.db

2. zot Push and Pull Image Content¶

Content Type	Client
OCI images	skopeo
OCI images	regclient (regctl)
OCI images	crane
OCI artifacts	oras
Helm charts	helm

3. Whether skopeo has skd or library

As skopeo pr 1891 said, we'll use github.com/containers/image
https://pkg.go.dev/github.com/containers/skopeo
https://pkg.go.dev/github.com/containers/image/v5/copy

4. Effects of zot and skopeo on sealer

1. During the build phase, the local image cache is used to speed up the build process, eliminating the need to pull an image from the remote registry every time
2. The run phase is the same as the existing registry. The performance difference is reflected in the difference between zot and registry itself
3. registry distribution follows existing rules with little performance change
4. For ha mode, you can continue to follow the existing implementation

5. The flow of the complete process in the container

5.1 Simulate running the sealer build/run process on a bare computer

Scene process:

1. buildah builds workingContainer based on an image and mounts it to filesystem (workingContainer is the predecessor of cluster image and rootfs is mounted).
2. skopeo connects to the local docker daemon to copy app images to workingContainer
3. buildah commit workingContainer to generate the final result image
4. Start the image and run zot to check whether the copy app images can be pulled successfully

Complete process record:

1. sealer build -f Kubefile -t skopeo-test:v1 --type app-installer

# Kubefile
From scratch

# related output
[root@k8s-master-32 ~]# sealer build -f Kubefile -t skopeo-test:v1 --type app-installer
STEP 1/2: FROM scratch
STEP 2/2: LABEL "io.sealer.version"="unknown"
COMMIT skopeo-test:v189d7ac4d
--> 2b6232da2b1
[Warning] one or more build args were not consumed: [TARGETARCH TARGETOS TARGETPLATFORM]
Successfully tagged localhost/skopeo-test:v189d7ac4d
2b6232da2b1efcb15401afff124f40e3d8158a7957a4a346d648e4880cc21b62
Getting image source signatures
Copying blob bd9ddc54bea9 skipped: already exists
Copying config dd52c49bae done
Writing manifest to image destination
Storing signatures
2023-02-03 16:00:20 [INFO] [build.go:445] image(linux/amd64) named as skopeo-test:v1, id: dd52c49baea761fe26c8a0e5ce6c41406aacdab713a5b80479c0a1c1399db948

2023-02-03 16:00:20 [INFO] [rmi.go:49] untagged: localhost/skopeo-test:v189d7ac4d

2023-02-03 16:00:20 [INFO] [rmi.go:54] 2b6232da2b1efcb15401afff124f40e3d8158a7957a4a346d648e4880cc21b62

[root@k8s-master-32 ~]# buildah images
REPOSITORY                      TAG            IMAGE ID       CREATED         SIZE
localhost/skopeo-test           v1             dd52c49baea7   2 minutes ago   3.2 KB

2. buildah from localhost/skopeo-test:v1

# related output
[root@k8s-master-32 ~]# buildah from localhost/skopeo-test:v1
skopeo-test-working-container

3. buildah mount skopeo-test-working-container

# related output
[root@k8s-master-32 ~]# buildah mount skopeo-test-working-container
/var/lib/containers/storage/overlay/bd974aa128ed71d04283a5e7b6eb5c8a875d53f3fb762c96eb0d79366027b693/merged

4. skopeo copy

# related output
[root@k8s-master-32 ~]# cd /var/lib/containers/storage/overlay/bd974aa128ed71d04283a5e7b6eb5c8a875d53f3fb762c96eb0d79366027b693/merged

[root@k8s-master-32 merged]# skopeo copy docker-daemon:busybox:latest oci:$(pwd)/oci-registry/busybox:v1
Getting image source signatures
Copying blob d6a7fc1fb44b done
Copying config bf86d23d3e done
Writing manifest to image destination
Storing signatures

[root@k8s-master-32 merged]# skopeo copy docker-daemon:alpine:3.13 oci:$(pwd)/oci-registry/alpine:3.13
Getting image source signatures
Copying blob 7df5bd7bd262 done
Copying config e0d2486fb1 done
Writing manifest to image destination
Storing signatures

[root@k8s-master-32 merged]# cd oci-registry/
[root@k8s-master-32 oci-registry]# tree
.
├── alpine
│   ├── blobs
│   │   └── sha256
│   │       ├── 05e07062ad2b305da1c3d3d8dda2388d6f4df948595dd85218c77007129eb9c3
│   │       ├── 986be8ca46800cd6b9c0e9b71f9e8a32b200a6e56d719e731536fc43a2ab5b7e
│   │       └── e0d2486fb1ef4e79e678c4cc950ac939d46fac4e4c38de5f467d29ba33a6b19e
│   ├── index.json
│   └── oci-layout
└── busybox
    ├── blobs
    │   └── sha256
    │       ├── 076588f6d6ab00f95c93be0211468a144e61ad67bb63969c682c9cd24b8c4656
    │       ├── 614a7d8579dab1b10b1d1b13fa15eef6929494dbba17284458057de9865f9486
    │       └── bf86d23d3e5c04d7232d4ae49d11af8dec39854c7bbcc83fe2571f8730a9fea8
    ├── index.json
    └── oci-layout

6 directories, 10 files

5. buildah commit

# related output
[root@k8s-master-32 oci-registry]# buildah unmount skopeo-test-working-container
5aaa0ca28d8b2f3e8251c7cc36879c2495da8bbda482944834e4b9e6468e1e28

[root@k8s-master-32 merged]# buildah commit skopeo-test-working-container skopeo-test:v1.0
Getting image source signatures
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d88e35bcc0fa done
Copying config 0c07ca5a58 done
Writing manifest to image destination
Storing signatures
0c07ca5a58b2138affdfd249f5dab679f63e8d4ab2f36396c3c1bb00a3f59308uhil

[root@k8s-master-32 merged]# sealer images
REPOSITORY                      TAG            IMAGE ID       CREATED              SIZE
localhost/skopeo-test           v1.0           0c07ca5a58b2   5 minutes ago        5.58 MB

6. sealer run localhost/skopeo-test:v1.0

# related output
[root@k8s-master-32 merged]# sealer run localhost/skopeo-test:v1.0
2023-02-03 16:21:59 [INFO] [pull.go:79] 0c07ca5a58b2138affdfd249f5dab679f63e8d4ab2f36396c3c1bb00a3f59308

2023-02-03 16:21:59 [INFO] [run.go:353] start to install application: localhost/skopeo-test:v1.0

2023-02-03 16:22:00 [INFO] [pull.go:79] 0c07ca5a58b2138affdfd249f5dab679f63e8d4ab2f36396c3c1bb00a3f59308

copying files to 10.10.101.34: 10/10
2023-02-03 16:22:05 [INFO] [run.go:412] succeeded in installing new app with image localhost/skopeo-test:v1.0

2023-02-03 16:22:06 [INFO] [remove_container.go:66] d2090f9d7b87c56adb63f486c652d54b25ef4c832350fc76c936ee7907c92427

# The content in rootfs is the same as that in workingContainer
[root@k8s-master-32 merged]# cd /var/lib/sealer/data/my-cluster/rootfs/oci-registry/
[root@k8s-master-32 oci-registry]# tree
.
├── alpine
│   ├── blobs
│   │   └── sha256
│   │       ├── 05e07062ad2b305da1c3d3d8dda2388d6f4df948595dd85218c77007129eb9c3
│   │       ├── 986be8ca46800cd6b9c0e9b71f9e8a32b200a6e56d719e731536fc43a2ab5b7e
│   │       └── e0d2486fb1ef4e79e678c4cc950ac939d46fac4e4c38de5f467d29ba33a6b19e
│   ├── index.json
│   └── oci-layout
└── busybox
    ├── blobs
    │   └── sha256
    │       ├── 076588f6d6ab00f95c93be0211468a144e61ad67bb63969c682c9cd24b8c4656
    │       ├── 614a7d8579dab1b10b1d1b13fa15eef6929494dbba17284458057de9865f9486
    │       └── bf86d23d3e5c04d7232d4ae49d11af8dec39854c7bbcc83fe2571f8730a9fea8
    ├── index.json
    └── oci-layout

6 directories, 10 files

7. docker run zot && docker pull

# related output
[root@k8s-master-32 ~]# docker run  -d -p 5050:5000 -v /var/lib/sealer/data/my-cluster/rootfs/oci-registry:/var/lib/registry ghcr.io/project-zot/zot-linux-amd64:v1.4.3
932919d78c0032b8e3fe7eae0061cfc187dc915e6104541c71900bf0450b7b4e

[root@k8s-master-32 ~]# docker pull localhost:5050/busybox:v1
v1: Pulling from busybox
076588f6d6ab: Pull complete
Digest: sha256:614a7d8579dab1b10b1d1b13fa15eef6929494dbba17284458057de9865f9486
Status: Downloaded newer image for localhost:5050/busybox:v1
localhost:5050/busybox:v1

[root@k8s-master-32 ~]# docker pull localhost:5050/alpine:3.13
3.13: Pulling from alpine
986be8ca4680: Pull complete
Digest: sha256:05e07062ad2b305da1c3d3d8dda2388d6f4df948595dd85218c77007129eb9c3
Status: Downloaded newer image for localhost:5050/alpine:3.13
localhost:5050/alpine:3.13

5.2 Run the sealer build procedure in the simulation container

Scene process:

1. Build an image whose container can run buildah and skopeo
2. buildah builds workingContainer based on an image and mounts it to filesystem (workingContainer is the predecessor of the cluster image and rootfs is mounted).
3. skopeo connects to the local docker daemon to copy app images to workingContainer
4. buildah commit workingContainer to generate an image of the final result
5. Start the image and run zot to check whether the copy app images can be pulled successfully

Complete process record:

1. Build an image whose container can run buildah and skopeo

# Dockerfile 
# skopeo-buildah:v1
FROM docker:dind

COPY sealer /usr/local/bin/
COPY kubernetes-v1.22.15.tar .

RUN apk update && \
    apk upgrade && \
    apk add bash && \
    apk add skopeo && \
    apk add buildah

ENTRYPOINT ["/bin/bash"]

2. sealer app image built in the container, i.e., repeat 5.1 Complete process 1-5

# related output
[root@k8s-master-32 skopeo-image]# docker run --privileged --name skopeo -it -v /var/run/docker.sock:/var/run/docker.sock skopeo-buildah:v1
fa7a21cd9cbe:/# sealer images
REPOSITORY              TAG    IMAGE ID       CREATED         SIZE
localhost/skopeo-test   v2.0   ba91933f70fa   8 seconds ago   5.57 MB
localhost/skopeo-test   v1     aba306c3095d   3 minutes ago   3.2 KB

3. Run the sealer image in the container
   In this step, you need to use kind. If the self-constructed image is used, port 22 cannot be connected.The above steps have proved that skopeo can copy the image outside the container and the constructed sealer image can normally perform the target function, so subsequent operation steps are omitted.

6. Code level design

6.1 Implement the following interface

// sealer/pkg/image/save/interface.go
type ImageSave interface {
	// SaveImages is not concurrently safe
	SaveImages(images []string, dir string, platform v1.Platform) error
	SaveImagesWithAuth(imageList ImageListWithAuth, dir string, platform v1.Platform) error 
}

6.2 saver Selecting a Mode

1. Add the sealer build parameter --imageSaver select saver, default to the old method
2. To build the image, you need to add annotations and specify the saver to use. In the run phase, you need to select the registry to start
3. Use different types of saver (kube-installer and app-installer use different saver)

• Simultaneous operation is not allowed
• Convert app images format using skopeo while running at the same time (affects sealer run speed)

[starnop]

Maybe it's not a good idea to add --imageSaver for sealer build, we should be discreet with each flag, especially with a core command like sealer build.

[jsparter]

Design update:

For compatibility and other requirements, the following scenarios needs to be concerned: (Use kube-installer and app-installer to directly express different types of SealerImages, and docker registry format and oci format indicate the app images store format in rootfs)

1. Both kube-installer and app-installer are use docker registry format
2. Both kube-installer and app-installer are use oci format
3. A kube-installer image had been installed with docker registry format, and the app-installer images with oci format will be installed into the cluster
4. A kube-installer image had been installed with oci format, and the app-installer images with docker registry format will be installed into the cluster

• Scenario 1 is what sealer is doing now.
• For Scenario 2, we need implement oci saver to load app images from docker-daemon and other place to rootfs in build stage, and modify init-registry.sh in rootfs. run stage needn't any change.
• For Scenario 3, there is a cluster created by sealer. We should identify format differences between app-installer and the cluster first by ImageExtension or other way, then we need a tool to transfer app-installer‘s oci-images to the running docker registry.
• Scenario 4 is a reverse version of the situation, just report an error.
  
  
  

[jsparter]

Remained to be finished:

• 1. basefs pr 40, modify basefs/auto-build.sh and basefs/context/download.sh to auto download zot-registry-image.tar.gz. Now I'm using ghcr.io/project-zot/zot-linux-amd64:v1.4.3，but it seems like there is no URL to get zot-registry-image.tar.gz yet.
• 2. multi-arch support
• 3. scenario 3, offer tool to push oci images to docker registry