Mirror registry is not working during bootstrap

[soulwhisper]

When I start a new talos cluster with private registry, it shows
"failed to pull image "gcr.io/etcd-development/etcd:v3.5.11": failed to resolve reference "gcr.io/etcd-development/etcd:v3.5.11": gcr.io/etcd-development/etcd:v3.5.11: not found",
however if I change the etcd image to "hub.homelab.internal/gcr.io/etcd-development/etcd:v3.5.11", it is working.

To reproduce my scenario, I will provide necessary snipplets below:

# talos-vmw-master-01.yaml
# 172.19.82.10 is my private dns, it contains record for hub.homelab.internal
...
  kubelet:
    image: ghcr.io/siderolabs/kubelet:v1.29.0
    extraArgs:
      rotate-server-certificates: "true"
    defaultRuntimeSeccompProfileEnabled: true
    disableManifestsDirectory: true
  network:
    hostname: talos-vmw-master-01
    interfaces:
      - deviceSelector:
          busPath: 0*
        addresses:
          - 172.19.82.31/24
        routes:
          - network: 0.0.0.0/0
            gateway: 172.19.82.1
        mtu: 1500
        dhcp: false
    nameservers:
      - 172.19.82.10
    disableSearchDomain: true
...
  registries:
    mirrors:
      docker.io:
        endpoints:
          - https://hub.homelab.internal/docker.io/
        overridePath: true
      gcr.io:
        endpoints:
          - https://hub.homelab.internal/gcr.io/
        overridePath: true
      ghcr.io:
        endpoints:
          - https://hub.homelab.internal/ghcr.io/
        overridePath: true
      registry.k8s.io:
        endpoints:
          - https://hub.homelab.internal/k8s.io/
        overridePath: true
    config:
      hub.homelab.internal:
        tls:
          insecureSkipVerify: true
        auth:
          username: <secret>
          password: <secret>

# my private registry is using Jfrog container registry, repo configs below
# all the registries are remote registry, work as a pull-through cache
# I have to use http-proxy to access some repo, so if you dont need it, just skip it

docker.io = https://registry-1.docker.io/, default
gcr.io = https://gcr.io/, disable 'blocking manifest v2 schema 1', w/ http-proxy, Bypass-HEAD-request
ghcr.io = https://ghcr.io/, default, w/ http-proxy
k8s.io = https://registry.k8s.io/, default, w/ http-proxy

# I have test direct pull w/ docker, daemon.json below
# both "docker pull hub.homelab.internal/ghcr.io/siderolabs/kubelet:v1.29.0" 
# and "docker pull hub.homelab.internal/gcr.io/etcd-development/etcd:v3.5.11" worked

{
  "live-restore": true,
  "dns": ["<my-internal-dns-ip>", "8.8.8.8"],
  "registry-mirrors": ["https://hub.homelab.internal"]
}

# note, hub.homelab.internal is enabled https w/ step-ca, its ca.crt is trusted by host with commands below
# this part is necessary for docker, yet talos using "insecureSkipVerify"
cp ca.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust

[soulwhisper]

using "talosctl dmesg | grep hub" will show talos try to search my domains in 1.1.1.1, which certainly get no answer, according to containerd/docs/hosts.md when hosts(mirror) of a server all fails, it will fall back to server url. so in the end it seems to be above error. correct me if I am wrong.

relative logs below

talosctl dmesg | grep retrying

172.19.82.31: user: warning: [2023-12-20T03:37:30.55778073Z]: [talos] retrying error: failed to pull image "ghcr.io/siderolabs/kubelet:v1.29.0": failed to resolve reference "ghcr.io/siderolabs/kubelet:v1.29.0": failed to do request: Head "https://hub.homelab.internal/ghcr.io/siderolabs/kubelet/manifests/v1.29.0?ns=ghcr.io": dial tcp: lookup hub.homelab.internal on 1.1.1.1:53: no such host
172.19.82.31: user: warning: [2023-12-20T03:37:30.55974773Z]: [talos] retrying error: failed to pull image "gcr.io/etcd-development/etcd:v3.5.11": failed to resolve reference "gcr.io/etcd-development/etcd:v3.5.11": failed to do request: Head "https://hub.homelab.internal/gcr.io/etcd-development/etcd/manifests/v3.5.11?ns=gcr.io": dial tcp: lookup hub.homelab.internal on 1.1.1.1:53: no such host

talosctl dmesg | grep hub

172.19.82.31: kern:    info: [2023-12-20T03:37:19.356164497Z]: usbcore: registered new interface driver hub
172.19.82.31: user: warning: [2023-12-20T03:37:30.644291497Z]: [talos] retrying error: failed to pull image "ghcr.io/siderolabs/kubelet:v1.29.0": failed to resolve reference "ghcr.io/siderolabs/kubelet:v1.29.0": failed to do request: Head "https://hub.homelab.internal/ghcr.io/siderolabs/kubelet/manifests/v1.29.0?ns=ghcr.io": dial tcp: lookup hub.homelab.internal on 1.1.1.1:53: no such host
172.19.82.31: user: warning: [2023-12-20T03:37:30.646258497Z]: [talos] retrying error: failed to pull image "gcr.io/etcd-development/etcd:v3.5.11": failed to resolve reference "gcr.io/etcd-development/etcd:v3.5.11": failed to do request: Head "https://hub.homelab.internal/gcr.io/etcd-development/etcd/manifests/v3.5.11?ns=gcr.io": dial tcp: lookup hub.homelab.internal on 1.1.1.1:53: no such host
172.19.82.31: user: warning: [2023-12-20T03:37:35.948464497Z]:  failed to pull image "ghcr.io/siderolabs/kubelet:v1.29.0": failed to resolve reference "ghcr.io/siderolabs/kubelet:v1.29.0": failed to do request: Head "https://hub.homelab.internal/ghcr.io/siderolabs/kubelet/manifests/v1.29.0?ns=ghcr.io": dial tcp: lookup hub.homelab.internal on 1.1.1.1:53: no such host
172.19.82.31: user: warning: [2023-12-20T03:37:35.954358497Z]:  failed to pull image "gcr.io/etcd-development/etcd:v3.5.11": failed to resolve reference "gcr.io/etcd-development/etcd:v3.5.11": failed to do request: Head "https://hub.homelab.internal/gcr.io/etcd-development/etcd/manifests/v3.5.11?ns=gcr.io": dial tcp: lookup hub.homelab.internal on 1.1.1.1:53: no such host

[soulwhisper]

@smira I have migrated my question from previous issue, and provide more info, please help me

[soulwhisper]

After a few retries, the solution seems clear.

• adding extraHostEntries failed, fallback happens before it applies
• manually replace bootstrap image url with mirrored url, w/ extraHostEntries, works
• there is a merge request about adding "disable fallback" at containerd, using private domains might need this before nameservers applies
• change mirror url from private domains to ip, confirmed working
• change mirror url from private domains to public domains, confirmed working

[frezbo]

you'd need /v2 for harbor, otherwise it's pointing to the harbor http UI

[soulwhisper]

It is not harbor. it is jfrog container registry.

[soulwhisper]

After a few retries, I found even if I managed to use mirror url to pull etcd / kubelet / kube-apiserver / pause, coredns still out of my reach. In kubectl describe pod coredns-85b955d87b-2xc9h -n kube-system, it shows

  Warning  FailedScheduling  27m (x11 over 77m)   default-scheduler  0/6 nodes are available: 6 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
  Normal   Scheduled         27m                  default-scheduler  Successfully assigned kube-system/coredns-85b955d87b-2xc9h to talos-vmw-master-03
  Warning  Failed            24m (x4 over 26m)    kubelet            Error: ErrImagePull
  Warning  Failed            24m                  kubelet            Failed to pull image "registry.k8s.io/coredns/coredns:v1.11.1": failed to pull and unpack image "registry.k8s.io/coredns/coredns:v1.11.1": failed to resolve reference "registry.k8s.io/coredns/coredns:v1.11.1": failed to do request: Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/coredns/coredns/manifests/v1.11.1": dial tcp 108.177.125.82:443: i/o timeout
  Warning  Failed            23m (x6 over 26m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff           12m (x48 over 26m)   kubelet            Back-off pulling image "registry.k8s.io/coredns/coredns:v1.11.1"
  Warning  Failed            7m26s (x6 over 26m)  kubelet            Failed to pull image "registry.k8s.io/coredns/coredns:v1.11.1": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "registry.k8s.io/coredns/coredns:v1.11.1": failed to resolve reference "registry.k8s.io/coredns/coredns:v1.11.1": failed to do request: Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/coredns/coredns/manifests/v1.11.1": dial tcp 108.177.125.82:443: i/o timeout

Seems mirror is not appied. Also, I wonder, if I install cilium later, will it pull from mirror?

[smira]

containerd only pulls images via CRI, e.g. the kubelet image is not pulled by containerd

but I think you should take a step back, figure out correct config (and I think /v2 should be somewhere in the path!), and things will just work as expected. we use registry mirrors a lot in our development and CI, so it's heavily tested.

JFrog might need its own quirks, but you can look into its logs to see where the problem is.

[soulwhisper]

I managed to switch from jfrog container registry to zot, then this issue is solved.