K0s configuration V2

[juanluisvaladas]

Is your feature request related to a problem? Please describe.

There a few concerns when it comes to k0s configuration. Some of these configuration are:
1- Lack of separation between node configuration and cluster configuration
2- Lack of feedback in dynamic config reconciliation status
3- Lack of feedback in helm and stack applier status (should these be in the clusterconfig to begin with?)
4- No separation between used defined configuration and the configuration that is used internally
5- Lack of versioning
6- Dynamic config has several problems when it comes to reconciliation of certain fields
7- Some fields have side effects that aren't very predictable.
8- The k0s' configuration for calico doesn't really match the calico configuration
9- Config doesn't allow to disable components, it requires modifying the cmdline, specially with dynamic config, it would make sense to allow this from the configuration.
10- In dynamicConfig it's not very straightforward what can and what cannot be modified

Describe the solution you would like

Ideally, everyone who uses k0s regularly will create an issue called config v2 proposal: <name> [optional short description] with the label config-2.0 a configuration file which describes what they would like to see.

Describe alternatives you've considered

No response

Additional context

No response

[dvirassamy]

The way where I struggled with customers are related to use-cases more than configuration :

1. How to do proper airgap deployments - there are multiple changes and ways of doing it, not very clear in doc
2. How to properly expose the cluster - again various ways of doing that and confusing
3. How to run day-2 operations, what is immutable and what is not is also not clear
4. DynamicConfig also raises confusion. What and what cannot be configured.

[dvirassamy]

Please also share how the migration will be handled with API deprecation and tooling to convert YAML config files from one version to another.

[ncopa]

I'd like to change the way we configure address family dualstack. Would be nice if we could:

• automatically enable address families that we provide CIDRs for (if you set both ipv4 and ipv6 CIDR we have dualstack).
• have some way to enable a list of address families (for using the default/autoetected values).

[kke]

1. I would consider removing all node-specific config keys from the yaml and move them to something like /etc/k0s/k0scontroller.conf or /etc/k0s/k0scontroller.service.d/storage.conf or similar.

k0s install would create a placeholder config that would look something like:

[Listen]
# API ports
apiPort=6443
k0sApiPort=9443

[Storage]
# Storage type (etcd, kine)
# storageType=etcd

[Network]
# ...
podCIDR=10.244.0.0/16
# ......
serviceCIDR=10.96.0.0/12

[Telemetry]
# enabled=true

[Users]
# Users for various components
#etcd=etcd
#kine=kube-apiserver
#konnectivity=konnectivity-server
#kubeAPIserver=kube-apiserver
#kubeScheduler=kube-scheduler
....

This config would also include not only the nodeconfig bits from current yaml but everything that can now be set via flags like [kubelet] extraArgs=.... Maybe all of the new node config values could also be also set via flags, something like --network-pod-cidr=10.244.0.0/16 or env K0S_NETWORK_POD_CIDR=10.244.0.0/16. Viper could probably handle most of this. The user should be able to change any of these settings and restart k0s to make them effective. If changing a value can break their existing cluster, it would be noted in the doc-comment.

2. Cluster-wide config keys would remain in the yaml. A default one could be placed somewhere as bootstrap.yaml during k0s install and it would be reconciled (or in non-dynamic copied to /var/lib/k0s/k0sworker.yaml or such?) upon first startup if a dynamic config can't be obtained from the peers. The bootstarp cluster config could also contain doc-comments, perhaps auto-generated from the tags/apidoc-comments. Changing a value that would break the cluster would make reconciler reject the config.