feature: k0s init, import some dependencies form k0sproject. #1619
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # k0s Runtime Design | ||
| ## basefs | ||
|
|
||
| ```shell | ||
| . | ||
| ├── amd64 | ||
| │ ├── bin | ||
| │ │ ├── k0s | ||
| │ │ ├── k0sctl | ||
| │ │ ├── kubectl | ||
| │ │ ├── nerdctl | ||
| │ │ └── seautil | ||
| │ ├── images | ||
| │ │ └── registry.tar.gz | ||
| │ └── Metadata | ||
| ├── imageList | ||
| ├── Kubefile | ||
| └── rootfs | ||
| ├── etc | ||
| │ ├── dump-config.toml | ||
| │ └── registry.yml | ||
| └── scripts | ||
| ├── containerd.sh | ||
| ├── init-registry.sh | ||
| └── init.sh | ||
| ``` | ||
|
|
||
| ## introduce | ||
| We define the k0s runtime have 5 phases to install/scale/reset the cluster. | ||
|
|
||
| basefs contain binary、shell script、config file and image. see more about [sealerio/basefs](https://github.com/sealerio/basefs) | ||
|
|
||
| Runtime before filesystem lead cluster install through execute [k0sctl](https://github.com/k0sproject/k0sctl) command. | ||
|
|
||
| + Init | ||
| + When sealer lead to cluster install first, init phase copy rootfs/bin to /usr/bin in init.sh script, make host ssh environment with no password, convert cluster file to k0sctl config, install the private registry. | ||
| + Join | ||
| + Join phase prepare the registry certs, and use `k0sctl apply` to reconcile the cluster. | ||
| + Delete | ||
| + Delete is same as join, but it recycles anything installed by join phase. | ||
| + Reset | ||
| + Reset through k0sctl to remove this cluster and remove anything about this cluster generated by sealer. | ||
| + Upgrade | ||
| + Upgrade can upgrade the k0s cluster. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| // Copyright © 2022 Alibaba Group Holding Ltd. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package k0s | ||
|
|
||
| const ( | ||
| GetK0sVersionCMD = "k0s version" | ||
|
|
||
| SSHKeyGenCMD = "ssh-keygen -f /root/.ssh/id_rsa -P \"\"" | ||
| SSHCopyIDPrefix = "ssh-copy-id -f " | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| // Copyright © 2022 Alibaba Group Holding Ltd. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package k0s | ||
|
|
||
| import ( | ||
| "fmt" | ||
| ) | ||
|
|
||
| func (k *Runtime) init() error { | ||
| pipeline := []func() error{ | ||
| k.MergeConfigOnMaster0, | ||
| k.SSHKeyGenAndCopyIDToHosts, | ||
| // TODO: move all these registry operation to the specific registry packages. | ||
| k.GenerateCert, | ||
|
There was a problem hiding this comment. This part of configuring private image repositories will be removed from the runtime pkg after we refactor run process, and i think k8s runtime also need those logic. see PR #1621 There was a problem hiding this comment. Got it! will adapt it when new code merged. There was a problem hiding this comment. I suggest that we do this before the code merging. Or at least you add a TODO comment |
||
| k.ApplyRegistryOnMaster0, | ||
| } | ||
|
|
||
| for _, f := range pipeline { | ||
| if err := f(); err != nil { | ||
| return fmt.Errorf("failed to prepare Master0 env: %v", err) | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| // MergeConfigOnMaster0 convert the cluster file spec to kubectl.yaml (k0sctl config file) to lead cluster run | ||
| func (k *Runtime) MergeConfigOnMaster0() error { | ||
| if err := k.K0sConfig.ConvertTok0sConfig(k.cluster); err != nil { | ||
| return fmt.Errorf("failed to convert to k0s config from clusterfile: %v", err) | ||
| } | ||
| ssh, err := k.getHostSSHClient(k.cluster.GetMaster0IP()) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to get ssh client: %v", err) | ||
| } | ||
| output, err := ssh.Cmd(k.cluster.GetMaster0IP(), GetK0sVersionCMD) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| k.K0sConfig.DefineConfigFork0s(string(output), k.RegConfig.Domain, k.RegConfig.Port, k.cluster.Name) | ||
|
|
||
| //write k0sctl.yaml to master0 rootfs | ||
|
There was a problem hiding this comment. I think the name of this func explicitly illustrates its ability. Maybe we don't have to annotate it. (It's ok to keep it.) |
||
| if err := k.K0sConfig.WriteConfigToMaster0(k.getRootfs()); err != nil { | ||
| return err | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // SSHKeyGenAndCopyIDToHosts use ssh-copy-id to prepare a no password ssh-client for k0sctl install environment. | ||
| func (k *Runtime) SSHKeyGenAndCopyIDToHosts() error { | ||
| return k.sshKeyGenAndCopyIDToHosts() | ||
| } | ||
|
|
||
| // GenerateCert generate the containerd CA for registry TLS. | ||
| func (k *Runtime) GenerateCert() error { | ||
| if err := k.GenerateRegistryCert(); err != nil { | ||
| return err | ||
| } | ||
| return k.SendRegistryCert(k.cluster.GetMasterIPList()[:1]) | ||
| } | ||
|
|
||
| // sshKeyGenAndCopyIDToHosts use ssh-key-gen and ssh-copy-id to prepare an env without password for k0sctl. | ||
| func (k *Runtime) sshKeyGenAndCopyIDToHosts() error { | ||
| if err := k.sshKeyGen(); err != nil { | ||
| return err | ||
| } | ||
| return k.sshCopyIDToEveryHost() | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| // Copyright © 2022 Alibaba Group Holding Ltd. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package k0sctl | ||
|
|
||
| const ( | ||
| K0sUploadBinaryPath = "/usr/bin/k0s" | ||
|
|
||
| ClusterFileRoleMaster = "master" | ||
| ClusterFileRoleWorker = "worker" | ||
|
|
||
| K0sController = "controller" | ||
| K0sWorker = "worker" | ||
| K0sControllerAndWorker = "controller+worker" | ||
|
|
||
| K0sDefaultUser = "root" | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| // Copyright © 2022 Alibaba Group Holding Ltd. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package k0sctl | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "strconv" | ||
|
|
||
| "github.com/sealerio/sealer/pkg/runtime/k0s/k0sctl/v1beta1" | ||
| v1 "github.com/sealerio/sealer/types/api/v1" | ||
| v2 "github.com/sealerio/sealer/types/api/v2" | ||
| utilsnet "github.com/sealerio/sealer/utils/net" | ||
| osi "github.com/sealerio/sealer/utils/os" | ||
|
|
||
| "github.com/k0sproject/dig" | ||
| "github.com/k0sproject/k0sctl/pkg/apis/k0sctl.k0sproject.io/v1beta1/cluster" | ||
| "github.com/k0sproject/rig" | ||
| yaml2 "gopkg.in/yaml.v2" | ||
| ) | ||
|
|
||
| type K0sConfig struct { | ||
| *v1beta1.Cluster | ||
| } | ||
|
|
||
| // ConvertTok0sConfig convert cluster file spec host to k0sctl spec hosts. | ||
| func (c *K0sConfig) ConvertTok0sConfig(clusterFile *v2.Cluster) error { | ||
| return c.convertIPVSToAddress(clusterFile) | ||
| } | ||
|
|
||
| func (c *K0sConfig) convertIPVSToAddress(clusterFile *v2.Cluster) error { | ||
|
There was a problem hiding this comment. I think the logic here seems to be parsing the host configuration information in the *v2.Cluster structure. Do we need to rename this function? There was a problem hiding this comment. This func mainly use sealer cluster file spec Hosts:IPS as input and out put k0s cluster spec Hosts:SSH:Address, so i named this. If there is a better way to rename it? I think though convertClusterHosts seems like convert some thing in Hosts field, but it don't represent we convert Hosts every field under Hosts. |
||
| masterIPList := utilsnet.IPsToIPStrs(clusterFile.GetIPSByRole(ClusterFileRoleMaster)) | ||
| if err := c.joinK0sHosts(masterIPList, clusterFile.Spec.Hosts, clusterFile.Spec.SSH, K0sController); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| nodeIPList := utilsnet.IPsToIPStrs(clusterFile.GetIPSByRole(ClusterFileRoleWorker)) | ||
| if err := c.joinK0sHosts(nodeIPList, clusterFile.Spec.Hosts, clusterFile.Spec.SSH, K0sWorker); err != nil { | ||
| return err | ||
| } | ||
| // TODO: Get the controller+worker role node. | ||
| // because sealer's cluster file do not support master and worker roles, | ||
| // so we can't generate a controller and worker role in k0s config. | ||
| return nil | ||
| } | ||
|
|
||
| func (c *K0sConfig) joinK0sHosts(ipList []string, hosts []v2.Host, ssh v1.SSH, role string) error { | ||
| for _, ip := range ipList { | ||
| port, err := c.parseSSHPortByIP(ip, hosts, ssh) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| //Now sealer do not support rootless, so default user is always 'root'. | ||
| c.addHostField(ip, port, role, K0sDefaultUser) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // addHostField generate k0s config spec Host and append it to config file. | ||
| func (c *K0sConfig) addHostField(ipAddr string, port int, role, user string) { | ||
| host := cluster.Host{ | ||
| Connection: rig.Connection{ | ||
| SSH: &rig.SSH{ | ||
| Address: ipAddr, | ||
| Port: port, | ||
| User: user, | ||
| }, | ||
| }, | ||
| Role: role, | ||
| UploadBinary: true, | ||
| K0sBinaryPath: K0sUploadBinaryPath, | ||
| } | ||
| c.Spec.Hosts = append(c.Spec.Hosts, &host) | ||
| } | ||
|
|
||
| // parseSSHPortByIP parse cluster file ssh port to k0s ssh port. | ||
| func (c *K0sConfig) parseSSHPortByIP(ipAddr string, hosts []v2.Host, ssh v1.SSH) (int, error) { | ||
| hostsMap := c.getHostsMap(hosts) | ||
| host := hostsMap[ipAddr] | ||
| if host.SSH.Port != "" { | ||
| return strconv.Atoi(host.SSH.Port) | ||
| } | ||
| return strconv.Atoi(ssh.Port) | ||
| } | ||
|
|
||
| // getHostsMap convert hosts to map[string]v1.SSH with ssh arg | ||
| func (c *K0sConfig) getHostsMap(hosts []v2.Host) map[string]v2.Host { | ||
| hostsMap := make(map[string]v2.Host) | ||
| for _, host := range hosts { | ||
|
There was a problem hiding this comment. The logic here should be able to simplify。 |
||
| ips := utilsnet.IPsToIPStrs(host.IPS) | ||
| for _, ip := range ips { | ||
| hostsMap[ip] = host | ||
| } | ||
| } | ||
| return hostsMap | ||
| } | ||
|
|
||
| // DefineConfigFork0s define the private registry for image repo such as: sea.hub:5000 | ||
| func (c *K0sConfig) DefineConfigFork0s(version, domain, port, name string) { | ||
| c.Spec.K0s = &cluster.K0s{ | ||
| Version: version, | ||
| Config: dig.Mapping{ | ||
| "apiVersion": "k0s.k0sproject.io/v1beta1", | ||
| "metadata": dig.Mapping{ | ||
| "name": name, | ||
| }, | ||
| "spec": dig.Mapping{ | ||
| "images": dig.Mapping{ | ||
| "repository": "\"" + domain + ":" + port + "\"", | ||
| }, | ||
| }, | ||
| }, | ||
| } | ||
| } | ||
|
|
||
| // WriteConfigToMaster0 write k0sctl config to rootfs | ||
| func (c *K0sConfig) WriteConfigToMaster0(rootfs string) error { | ||
| k0sConfig := c.Cluster | ||
| marshal, err := yaml2.Marshal(k0sConfig) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if err := osi.NewAtomicWriter(rootfs + "/k0sctl.yaml").WriteFile(marshal); err != nil { | ||
| return fmt.Errorf("failed to write k0sctl config: %v ", err) | ||
| } | ||
| return nil | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not very familiar with k0s, I have a question, is this necessary to configure ssh key login？
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we don't ssh-keygen id_rsa config: when k0sctl lead to install cluster, it will report:
ERRO [ssh] 172.16.161.63:22: attempt 1 of 60.. failed to connect: stat /root/.ssh/id_rsa: no such file or directoryif we don't ssh-copy-id: k0sctl will report :
failed to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain