kubelet client certificate ended up empty! how to recover?

[danmilon]

Hello,

I'm not very knowledgeable about the low level components of kubernetes and how they fit together and auth so please bear with me if I'm talking nonsense.

On a talos linux home server (single node k8s cluster) I realized that things were down and upon inspection the error seems to be that kubelet is loop crashing with the following error:

bmax.internal: {"ts":1706909997906.9836,"caller":"app/server.go:487","msg":"Kubelet version","v":0,"kubeletVersion":"v1.29.0"}
bmax.internal: {"ts":1706909997907.0105,"caller":"app/server.go:489","msg":"Golang settings","v":0,"GOGC":"","GOMAXPROCS":"","GOTRACEBACK":""}
bmax.internal: {"ts":1706909997907.2651,"caller":"app/server.go:919","msg":"Client rotation is on, will bootstrap in background","v":0}
bmax.internal: {"ts":1706909997908.01,"caller":"bootstrap/bootstrap.go:256","msg":"unable to load TLS certificates from existing bootstrap client config read from /etc/kubernetes/kubeconfig-kubelet: data does not contain any valid RSA or ECDSA certificates"}
bmax.internal: {"ts":1706909997909.0054,"caller":"certificate/certificate_store.go:130","msg":"Loading cert/key pair from \"/var/lib/kubelet/pki/kubelet-client-current.pem\".","v":0}
bmax.internal: {"ts":1706909997909.0881,"caller":"cli/run.go:74","msg":"command failed","err":"failed to run Kubelet: failed to initialize client certificate manager: could not convert data from \"/var/lib/kubelet/pki/kubelet-client-current.pem\" into cert/key pair: tls: failed to find any PEM data in certificate input"}

Indeed, the symlink target is an empty file:

danmilon@delogo ~/talos-configs/recovery $ talosctl -e bmax.internal -n bmax.internal list -l /var/lib/kubelet/pki/
NODE            MODE         UID   GID   SIZE(B)   LASTMOD           NAME
bmax.internal   drwxr-xr-x   0     0     124       Feb  2 23:05:49   .
bmax.internal   -rw-------   0     0     0         Feb  2 21:01:33   kubelet-client-2024-02-02-19-01-33.pem
bmax.internal   Lrwxrwxrwx   0     0     59        Feb  2 21:01:33   kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2024-02-02-19-01-33.pem
bmax.internal   -rw-r--r--   0     0     2242      Feb  2 23:05:49   kubelet.crt
bmax.internal   -rw-------   0     0     1679      Feb  2 23:05:49   kubelet.key

First, I'm not sure how it ended up like that really. Is there anything I can do to figure out?

Second, is there a way to recover things?

Thank you!

[smira]

Hi, at this moment the only way to get out of this case might be to wipe the worker node's EPHEMERAL partition with:

talosctl reset -n WORKER_NODE --system-labels-to-wipe=EPHEMERAL --reboot

Please not this wipes the whole /var partition.

This will force the kubelet to re-join.

The root cause is most probably failed write by the kubelet. Talos 1.7 plans to improve on the recovery procedure in this case.

[danmilon]

Thanks a lot! I think some files related to mayastor are stored somewhere in /var, although in the machine config I've set up a bind mount for that dir, not sure if it'd still be lost. Alternatively I was thinking to take out the node's disk and manually create an apiserver client certificate and update the file. Do you think that'd work as well?

…

On Sun, 4 Feb 2024, 12:38 Andrey Smirnov, ***@***.***> wrote: Hi, at this moment the only way to get out of this case might be to wipe the worker node's EPHEMERAL partition with: talosctl reset -n WORKER_NODE --system-labels-to-wipe=EPHEMERAL --reboot Please not this wipes the whole /var partition. This will force the kubelet to re-join. The root cause is most probably failed write by the kubelet. Talos 1.7 plans to improve on the recovery procedure in this case. — Reply to this email directly, view it on GitHub <#8248 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIWHC5RAVFYVHRMHIOPNH3YR5QKXAVCNFSM6AAAAABCXK2XMSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGNJZHAYTK> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[smira]

if you have direct access to the node's disk, you can clean up those files in /var/lib/kubelet/pki and the kubelet should re-join