Switch to openSUSE MicroOS "k3s"

[mysticaltech]

Hey folks, Rancher got sold to SUSE, and it seems that SUSE won't support this project because it does not use a SUSE distro.

So only one logical step to take if we want this project to survive and projects that depend on it is to move it to k3s-io which is an independent org. What do you think?

[mysticaltech]

@dweomer Please, can you make this happen? Otherwise, we're all screwed (anyone that relies on k3os), like my dear little tiny project kube-hetzner.

[srgvg]

Weird, I was expecting Suse to bring exciting stuff here?
https://twitter.com/ibuildthecloud/status/1375878115726041089

[mysticaltech]

Sadly, k3os does not seem on the menu!

[Rtoax]

Sadly, k3os does not seem on the menu!

OMG

[thejimnicholson]

I'm in favor of this, and frankly, if it doesn't move to k3s-io, I'd still favor forking it, and creating its own org if necessary.

[mysticaltech]

Definitely, that's the only way @thejimnicholson!

Otherwise, we could also try to piggyback on container OS like AWS's Bottlerocket and integrate k3s in it, along with making sure that the system upgrade controller works.

But that would be a shame, as k3os already does all of these things, it just needs continuous support and maintenance.

[Rtoax]

I'm in favor of this, and frankly, if it doesn't move to k3s-io, I'd still favor forking it, and creating its own org if necessary.

Cool

[mysticaltech]

Have a look at how we're done it over at Kube-Hetzner; we no longer need k3os! https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner

In short, we install k3s on top of OpenSuse MicroOS!

[Ludea]

There are https://github.com/rancher-sandbox/cOS-toolkit and https://github.com/harvester/os2 (which is based on cOS-toolkit)

[mysticaltech]

Thanks for sharing, cOS-toolkit seems interesting! https://rancher-sandbox.github.io/cos-toolkit-docs/docs/tutorials/k3s_and_fleet_on_vanilla_image_example/

[BoscaBeag]

Like everyone else here, I'd like to know what the future holds. k3os is the best/cleanest way I've found to bring up a k8s instance with the least amount of operating system admin-ing. (Having to build my own ISOs is even deeper down that rabbit hole.)

[mysticaltech]

Thanks, @Ludea for pointing this out. cOS-toolkit seems to be the replacement for k3os folks. It's under heavy dev and seems flexible enough, with more or less the same flow we had with k3os, but it's just a tiny bit more generic.

If anyone has the courage to test it here's a guide for k3s and system upgrade controller, it would be good to be sure that it can deliver a k3os-like experience!

[mudler]

Just sharing here my notes.. as a k3os user.

I've been using k3os as well for long time and I've been a happy user since with. cos-toolkit does indeed provide a foundation to write derivatives similarly to k3os but allows more fine grained control on the image on the system (if someone would want to), so there you would find a lot of common parts with k3os philosophy.

If you are interested in building/running a "k3os" variant based on cos-toolkit, feel free to have a look here, it's one way to consume the toolkit to produce a derivative. I'm really interested in any feedback at this stage.

[mysticaltech]

Thanks for sharing @mudler! Really good to see what you've done. Have you explored OpenSuse MicroOS? Seems really good, IMHO. If that can be integrated with cOS-toolkit, and work well with the system-upgrade-controller, we're good to go!

[mysticaltech]

@phaer Adding you to that discussion since you have been spending some time on kube-hetzner, you deserve to know, that we may need to change the underlying OS soon.

It's doable, just trying to find out the best course of action. Before was using Fedora Server, shifted to k3os, and maybe will need to move to OpenSuse MicroOS. It's a WIP.

[philipsparrow]

If SuSE has a good solution for a simple OS that runs K3s (MicroOS?), a solution to upgrade from K3OS to the SuSE solution would be a great way to move clusters over

[mysticaltech]

Yes! Exactly @philipsparrow. If you are curious, for Kube-Hetzner, we already started the find the path to migrate to MicroOS. See kube-hetzner/terraform-hcloud-kube-hetzner#35.

[philipsparrow]

Thanks @mysticaltech that's really interesting. Forgive me for not following the discussion fully, but does the dd writing allow the writing of MicroOS to the K3OS disk via the system-upgrade-controller? So I could upgrade to MicroOS remotely? That would be amazing.

[mysticaltech]

Hello @philipsparrow, unfortunately, no it does not allow that. DD is an option by we found out another way in kube-hetzner/terraform-hcloud-kube-hetzner#35, it's a huge thread but basically, we used this

MICROOS_DISK="https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-k3s-kvm-and-xen.qcow2"
curl -sL "${MICROOS_DISK}" -o /tmp/MicroOS.qcow2
qemu-img convert -p -f qcow2 -O host_device /tmp/MicroOS.qcow2 /dev/sda

And after that, we create a partition on the disk to host the ignition file to input our SSH key.

If you have an active cluster, what you could do is drain the node, deploy a new node with MicroOS "k3s", join it to the cluster with the token and the right config, etc, and repeat for all nodes. Never done that, but theoretically, it should work. Good luck!

[mysticaltech]

Folks, just so you know, turns out MicroOS is just great. And they have a release that ships with k3s already configured with https://build.opensuse.org/package/show/openSUSE:Factory/k3s.

You can find it here https://mirrorcache.opensuse.org/tumbleweed/appliances/, that is what I use.

However, you could also theoretically get a "vanilla" MicroOS from the MicroOS download page, and install the k3s package as follows (did not test): transactional-updates pkg install k3s.

If you go into the k3s package, you will see how to configure it, I personally just ignored SERVER_OPTS or AGENT_OPTS, and in both situations "server" or "agent", and just created a k3s config in /etc/rancher/k3s/config.yaml (for each) that had all the options I needed.

When configured as above, you just do, either:
systemctl --now enable k3s-server or systemctl --now enable k3s-agent

And initially, to add your SSH key on boot and configure third-party networks, everything can be done with a system called ignition, basically, you need a partition labeled "ignition", anywhere, USB, eternal volume, or even on the same disk, with a folder ignition in to, and a config.ign file. Very simple when you understand how to do that. See examples of how we did it over at kube-hetzner/terraform-hcloud-kube-hetzner#35. One thing to know is that the "SelfInstall" images, do not have that, I believe they have a GUI for setup.

Everything just works great, it even does automatic upgrades without the need for the system upgrade controller, thanks to MicroOS transactional upgrades system.

Basically, it's k3os, but better! Good luck :)

[srgvg]

just happened to bump into this project, uses c3os for k3s https://github.com/mudler/c3os

[alexdrl]

@mysticaltech is there some way to somehow upgrade from a k3os installation to a MicroOS deployment in place?

[mysticaltech]

Completely doable! You need the right time of node, server or agent, the same k3s config, the server url (of one of you control plane nodes) and the token.

You joint a few microOS k3s nodes control planes and agents, then drain and disconnect the k3os ones.

If you are curious how to install k3s on microOS, it's super easy when you understand it, check out how we do it at https://github.com/kube-hetzner/kube-hetzner

[bkraegelin]

I have a problem, it just does not want to work.

I installed https://mirrorcache.opensuse.org/tumbleweed/appliances/iso/openSUSE-MicroOS.x86_64-k3s-SelfInstall.iso as my OS distribution. After installation I resized the disk according to kube-hetzner/terraform-hcloud-kube-hetzner#35 (comment) and then I started k3s-server according to #838 (comment) with /etc/rancher/k3s/server.conf containing SERVER_OPTS="--cluster-init"

First view looks good, I even can attach additional workes. Second look is bad kubectl get pods -A shows all pods with status ContainerCreating. No progress even on the next day.

Something is missing, but I don't know. Any help?

Thanks,
Birger

[mysticaltech]

@bkraegelin Why are you doing this? Just use https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner to deploy yourself a single node Kube cluster. It will do everything correctly for you!

[bkraegelin]

We just need to know, HOW to do this. We need to find our own automation solution to deploy about 100 clusters all around Germany. To run local On Prem services implemented as kubernetes pods and centrally managed using Rancher UI. Platform will be bare metal.

[mysticaltech]

The solution is not easily explainable in text. It's quite complex with multiple layers. Just have a look at our repo above, to see how we do it on Hetzner.

[bish0polis]

I've come here late, following a thread from another discussion.

I'm sad to see it's toast. You see, I worked on UnitedLinux, and my choices are forever simplified by one from the experience.

I'll be looking for a successor project to recommend.

[kyeotic]

Did you find anything @bish0polis? I've seen a few options, but none half as simple as k3os was.

[dweomer]

I no longer work at Rancher/SUSE, close to a year now, and have no affiliation with the following recommendation, but Ettore tells me that https://kairos.io/ has influence from k3os. Maybe it could be a natural successor for your use case(s)?