Skip to content

Latest commit

 

History

History
1070 lines (792 loc) · 39.4 KB

UPGRADE.md

File metadata and controls

1070 lines (792 loc) · 39.4 KB
Raw

This document guides you through the process of upgrading Kuma.

First, check if a section named Upgrade to x.y.z exists, with x.y.z being the version you are planning to upgrade to.

If such a section does not exist, the upgrade you want to perform does not have any particular instructions.

Upgrade to 2.4.x

Configuration change

The configuration: Metrics.Mesh.MinResyncTimeout and Metrics.Mesh.MaxResyncTimeout are replaced by Metrics.Mesh.MinResyncInterval and Metrics.Mesh.FullResyncInterval. You can still use the current configs but it will be removed in the future.

Breaking changes

Removal of service field in Dataplane outbound

After a period of depreciation, the service field in now removed. The service name is only defined by the value of kuma.io/service in the outbound tags field.

Upgrade to 2.3.x

Breaking changes

MeshHTTPRoute

  • Changed path match type from Prefix to PathPrefix

MeshAccessLog

MeshTrace

kumactl container image

  • Changed image's entrypoint to /usr/bin/kumactl

This change was introduced to be consistent with kuma-cp and kuma-dp images, where names of images refer to binaries set in entrypoint.

Example valid before:

docker run kumahq/kumactl:2.2.1 kumactl install transparent-proxy --help

Equivalent example valid now:

docker run kumahq/kumactl:2.3.0 install transparent-proxy --help

TLS verification between Zone CP and Global CP

If the CA used to sign the Global CP sync server is not provided to a Zone CP (HELM controlPlane.tls.kdsZoneClient, ENV: KUMA_MULTIZONE_ZONE_KDS_ROOT_CA_FILE), and the certificate is signed by a CA that is not included in the system's CA bundle on the Zone CP machine, you must do one of the following:

Removal of Common Name from generated certificates

This only affects users who rely on generated certificates having a common name set.

  • kumactl generate tls-certificate generates certificates without CN
  • autogenerated TLS certificate for kuma-cp (when general.tlsCertFile is not provided) won't have CN

Upgrade to 2.2.x

Universal

CentOS 7

We are dropping support for running Envoy on CentOS 7 with this release and will not release CentOS 7 compatible Envoy builds.

Changed default postgres driver to pgx

  • If you encounter any problems with the persistence layer please submit an issue and temporarily switch to the previous driver (lib/pq) by setting DriverName=postgres configuration option or KUMA_STORE_POSTGRES_DRIVER_NAME='postgres' env variable.
  • Several configuration settings are not supported by the new driver right now, if used to configure them please try running with new defaults or submit an issue. List of unsupported configuration options:
    • MaxIdleConnections (used in store)
    • MinReconnectInterval (used in events listener)
    • MaxReconnectInterval (used in events listener)

Longer name of the resource in postgres

Kuma now permits the creation of a resource with a name of up to 253 characters, which is an increase from the previous limit of 100 characters. This adjustment brings our system in line with the naming convention supported by Kubernetes. This change requires to run kuma-cp migrate up to apply changes to the postgres database.

K8s

Removed deprecated annotations

  • kuma.io/builtindns and kuma.io/builtindnsport are removed in favour of kuma.io/builtin-dns and kuma.io/builtin-dns-port introduced in 1.8.0. If you are using the legacy CNI you main need to set these old annotations manually in your pod definition.
  • kuma.io/sidecar-injection is no longer supported as an annotation, you should use it as a label.

Helm

All containers now have defaults for resources.requests.{cpu,memory} and resources.limits.{memory}. There are new default values for *.podSecurityContext and *.containerSecurityContext, see values.yaml.

Gateway API

We now support version v0.6.0 of the Gateway API. See the upstream API changes for more info.

Auth configuration of DP server in Kuma CP

dpServer.auth configuration of Kuma CP was deprecated. You can still set config in this section, but it will be removed in the future. It's recommended to migrate to dpServer.authn if you explicitly set any of the configuration in this config section.

  • dpServer.auth.type is now split into two: dpServer.authn.dpProxy.type and dpServer.authn.zoneProxy.type and is still autoconfigured based on the environment.
  • dpServer.auth.useTokenPath is now dpServer.authn.enableReloadableTokens

Transparent Proxy Engine v2 and CNI v2 as default

As they matured, in the upcoming release Kuma will by default use transparent proxy engine v2 and CNI v2.

If you want to still use v1 versions of these components, you will have to install Kuma with provided legacy.transparentProxy=true or legacy.cni.enabled=true options.

Examples

CNI

Helm

helm upgrade --install --create-namespace --namespace kuma-system \
  --set "legacy.cni.enabled=true" \
  --set "cni.enabled=true" \
  --set "cni.chained=true" \
  --set "cni.netDir=/etc/cni/net.d" \
  --set "cni.binDir=/opt/cni/bin" \
  --set "cni.confName=10-calico.conflist"
  kuma kuma/kuma

kumactl

kumactl install control-plane \
  --set "legacy.cni.enabled=true" \
  --set "cni.enabled=true" \
  --set "cni.chained=true" \
  --set "cni.netDir=/etc/cni/net.d" \
  --set "cni.binDir=/opt/cni/bin" \
  --set "cni.confName=10-calico.conflist" \
  | kubectl apply -f-
Transparent Proxy Engine

Helm

helm upgrade --install --create-namespace --namespace kuma-system \
  --set "legacy.transparentProxy=true" kuma kuma/kuma

kumactl

kumactl install control-plane --set "legacy.transparentProxy=true" | kubectl apply -f-

Removal of deprecated options to reach applications bound to localhost

The deprecated options KUMA_DEFAULTS_ENABLE_LOCALHOST_INBOUND_CLUSTERS and defaults.enableLocalhostInboundClusters were removed.

This change affects only applications using transparent proxy.

Applications that are binding to localhost won't be reachable anymore. This is the default behaviour from Kuma 1.8.0. Until now, it was possible to set a deprecated kuma-cp configurations KUMA_DEFAULTS_ENABLE_LOCALHOST_INBOUND_CLUSTERS or defaults.enableLocalhostInboundClusters to true, which was allowing to still reach these applications.

One of the options to upgrade change address which the application is listening on, to 0.0.0.0. Other option is to define dataplane.networking.inbound[].serviceAddress to the address which service is binding to.

Upgrade to 2.1.x

Breaking changes

Naming Serviceless dataplanes has changed

Currently, the kuma.io/service value of the inbound of a Dataplane generated for a Pod without a Service is based on the Pod name. The Kuma CP takes the pod's name and removes 2 last elements after splitting by -. This behavior is correct when the Pod is owned by a Deployment or CronJob but not for other owner kinds. Kuma will now use the name of the owner resource as the kuma.io/service value. Before upgrade:

  1. Identify all Serviceless Pods that are not managed by a Deployment or CronJob.
  2. Create copies of policies that were created for the services corresponding to these Pods. The kuma.io/service value is the name of the owner resource. If there is no owner, Kuma uses the Pod's name.

This breaking change is required to provide correct naming. The previous behavior could produce the same kuma.io/service value of the inbound of a Dataplane for many different serviceless Dataplanes.

MeshTrafficPermission

Action value have switched to PascalCase. ALLOW is Allow, DENY is Deny and ALLOW_WITH_SHADOW_DENY is AllowWithShadowDeny.

HTTP api

We've removed the deprecated endpoint POST /tokens, use the POST /tokens/dataplane endpoint instead (same request and response). Make sure you are using a recent kumactl or that you use the right path if using the API directly to upgrade with no issues.

Kubernetes

The sidecar container is always injected first (since #5436). This should only impact you when modifying the sidecar container with a container-patch. If you do so, upgrade Kuma and then change your container patch to modify the right container.

This version changes the leader election mechanism from leader for life to the more robust leader with lease. As the result, during the upgrade you may have two leaders in the cluster. This should not impact the system in any significant way other than logs like resource was already updated.

Kumactl

--valid-for must be set for all token types, before it was defaulting to 10 years.

Upgrade to 2.0.x

Built-in gateway

If you're using the PREFIX path match for MeshGatewayRoute, note that validation is now stricter. If you try to update an existing MeshGatewayRoute or create a new one, make sure your PREFIX matching value does not include a trailing slash. All prefix matches are checked path-separated, meaning that /prefix only matches if the request's path is /prefix or begins with /prefix/. This has always been the case, so no behavior has been changed and existing resources with a trailing slash are not affected.

Universal

A lib/pq change enables SNI by default when connecting to Postgres over TLS. Either make sure your certificates contain a valid CN or SANs for the hostname you're using or update to 2.0.1 and disable sslsni by setting the KUMA_STORE_POSTGRES_TLS_DISABLE_SSLSNI environment variable or store.postgres.tls.disableSSLSNI in the config to true.

kuma-prometheus-sd

This component has been removed after a long period of deprecation.

Zone Ingress Token migration

This is only relevant to Multizone deployment with Universal zones. Zone Token that was previously used for authenticating Zone Egress, can now be used to authenticate Zone Ingress. Please regenerate Zone Ingress token using kumactl generate zone-token --scope=ingress. For the time being you can still use the old Zone Ingress token and Zone Token with scope ingress. However, Zone Ingress Token is now deprecated and will be removed in the future.

Helm

ingress.annotations and egress.annotations are deprecated in favour of ingress.podAnnotations and egress.podAnnotations which is a better name and aligne with the existing controlPlane.podAnnoations.

Kuma-cp

  • By default, the minimum TLS version allowed on servers is TLSv1.2. If you require using TLS < 1.2 you can set KUMA_GENERAL_TLS_MIN_VERSION.
  • KUMA_MONITORING_ASSIGNMENT_SERVER_GRPC_PORT was removed after a long deprecation period use KUMA_MONITORING_ASSIGNMENT_SERVER_PORT instead.

gRPC metrics

With this release, emitting separate statistics for every gRPC method is disabled. gRPC metrics from different methods are now aggregated under envoy_cluster_grpc_request_message_count. It will be re-enabled again in the future once Envoy with replace_dots_in_grpc_service_name feature is released. If you need to enable this setting, you can use ProxyTemplate to patch envoy.filters.http.grpc_stats http filter.

Upgrade to 1.8.x

Kumactl

  • kumactl inspect dataplane --config-dump was deprecated in favour of kumactl inspect dataplane --type config-dump. The behaviour of the new flag is unchanged but you should migrate.
  • kumactl install transparent-proxy --skip-resolv-conf was deprecated as there's no reason for us to update the /etc/resolv.conf of the user.
  • kumactl install transparent-proxy --kuma-cp-ip was removed as it's not possible to run a DNS server on the cp.

Helm

  • Under cni.image, the default values for repository and registry have been changed to agree with the other image values.

CP

  • The /versions endpoint was removed. This is not something that was reliable enough and version compatibility is checked inside the DP
  • We are deprecating kuma.io/builtindns and kuma.io/builtindnsport annotations in favour of the clearer kuma.io/builtin-dns and kuma.io/builtin-dns-port. The behavior of the new annotations is unchanged but you should migrate (a warning is present on the log if you are using the deprecated version).
  • By default, applications binding to localhost are not reachable anymore. A Dataplane inbound's default serviceAddress is now the inbound's address. Before upgrade, if you have applications listening on localhost that you want to expose on:
    • Kubernetes: listen on 0.0.0.0 instead
    • Universal: listen on inbound.address instead or set dataplane.networking.inbound[].serviceAddress: "127.0.0.1" To make migration easier you can temporarily disable this new behavior by setting KUMA_DEFAULTS_ENABLE_LOCALHOST_INBOUND_CLUSTERS=true on kuma-cp, this option will be removed in a future version.

Upgrade to 1.7.x

Kumactl

  • We're deprecating kumactl install metrics/tracing/logging, please use kumactl install observability instead

DNS

The control-plane no longer hosts a builtin DNS server. You should always rely on the embedded DNS in the dataplane proxy and VIPs can't be used without transparent proxy.

Timeout policy

'grpc' section is deprecated. Timeouts for HTTP, HTTP2 and GRPC should be set in 'http' section:

tcp: 
  idleTimeout: 1h 
http: # http, http2, grpc
  requestTimeout: 15s 
  idleTimeout: 1h
  streamIdleTimeout: 30m
  maxStreamDuration: 0s
grpc: # DEPRECATED
  streamIdleTimeout: 30m # DEPRECATED, use 'http.streamIdleTimeout'
  maxStreamDuration: 0s # DEPRECATED, use 'http.maxStreamDuration'

Upgrade to 1.6.x

Helm

  • the Helm chart for this release requires at least Helm version 3.8.0.
  • controlPlane.resources is now on object instead of a string. Any existing value should be adapted accordingly.

Zone egress and ExternalService

When an ExternalService has the tag kuma.io/zone and ZoneEgress is enabled then the request flow will be different after upgrading Kuma to the newest version. Previously, the request to the ExternalService goes through the ZoneEgress in the current zone. The newest version flow is different, and when ExternalService is defined in a different zone then the request will go through local ZoneEgress to ZoneIngress in zone where ExternalService is defined and leave the cluster through ZoneEgress in this cluster. To keep previous behavior, remove the kuma.io/zone tag from the ExternalService definition.

Zone egress

Previously, when mTLS was configured and ZoneEgress deployed, requests were routed automatically through ZoneEgress. Now it's required to explicitly set that traffic should be routed through ZoneEgress by setting Mesh configuration property routing.zoneEgress: true. The default value of the property is set to false so in case your network policies don't allow you to reach other external services/zone without using ZoneEgress, set routing.zoneEgress: true.

type: Mesh
name: default
mtls: # mTLS is required for zoneEgress
 [...]
routing:
 zoneEgress: true

The new approach changes the flow of requests to external services. Previously when there was no instance of ZoneEgress traffic was routed directly to the destination, now it won't reach the destination.

Gateway (experimental)

Previously, a MeshGatewayInstance generated a Deployment and Service whose names ended with a unique suffix. With this release, those objects will have the same name as the MeshGatewayInstance.

Inspect API

In connection with the changes around MeshGateway and MeshGatewayRoute, the output schema of the <policy-type>/<policy>/dataplanes has changed. Every policy can now affect both normal Dataplanes and Dataplanes configured as builtin gateways. The configuration for the latter type is done via MeshGateway resources.

Every item in the items array now has a kind property of either:

  • SidecarDataplane: a normal Dataplane with outbounds, inbounds, etc.
  • MeshGatewayDataplane: a MeshGateway-configured Dataplane with a new structure representing the MeshGateway it serves.

Some examples can be found in the Inspect API docs.

Upgrade to 1.5.x

Any type

The kuma.metrics.dataplane.enabled and kuma.metrics.zone.enabled configurations have been removed.

Kuma always generate the corresponding metrics.

Kubernetes

  • Please migrate your kuma.io/sidecar-injection annotations to labels. The new version still supports annotation, but to have a guarantee that applications can only start with sidecar, you must use label instead of annotation.
  • Configuration parameter kuma.runtime.kubernetes.injector.sidecarContainer.adminPort and environment variable KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_ADMIN_PORT have been deprecated in favor of kuma.bootstrapServer.params.adminPort and KUMA_BOOTSTRAP_SERVER_PARAMS_ADMIN_PORT.

Universal

  • We removed support for old Ingress (Dataplane#networking.ingress) from pre 1.2 days. If you are still using it, please migrate to ZoneIngress first (see Upgrade to 1.2.0 section).
  • You can't use 0.0.0.0 or :: in networking.address most of the time using loopback is what people intended.
  • Kuma DP flag --admin-port and environment variable KUMA_DATAPLANE_ADMIN_PORT have been deprecated, admin port should be specified in Dataplane or ZoneIngress resources.

Upgrade to 1.4.0

Starting with this version, the default API server authentication method is user tokens. In order to continue using client certificates (the previous default method), you'll need to explicitly set the authentication method to client certificates. This can be done by setting the KUMA_API_SERVER_AUTHN_TYPE variable to "clientCerts".

See Configuration - Control plane for how to set this variable.

Upgrade to 1.3.0

Starting with this version Mesh resource will limit the maximal number of mtls backends to 1, so please make sure your Mesh has correct backend applied before the upgrade.

Outbound generated internally are no longer listed in dataplane.network.outbound[]. For Kubernetes, they will automatically disappear. For universal to remove them you should recreate your dataplane resources (either with kumactl apply or by restarting your services if the dataplanes lifecycle is managed by Kuma).

Kuma 1.3.0 has additional mechanism for tracking data plane proxies and zone statuses in a more reliable way. This mechanism works as a heartbeat and periodically increments the generation counter for the Insights. If the overall time for upgrading all Kuma CP instances is more than 5 minutes, then some data plane proxies or zones may become Offline in the GUI, but this doesn't affect real connectivity, only view. This unwanted effect will disappear as soon as all Kuma CP instances will be upgraded to 1.3.0.

Upgrade to 1.2.1

When Global is upgraded to 1.2.1 and Zone CP is still 1.2.0, ZoneIngresses will always be listed as offline. After Zone CPs are upgraded to 1.2.1, the status will work again. ZoneIngress status does not affect cross-zone traffic.

Upgrade to 1.2.0

One of the changes introduced by Kuma 1.2.0 is renaming Remote Control Planes to Zone Control Planes and Dataplane Ingress to Zone Ingress. We think this change makes the naming more consistent with the rest of the application and also removes some of unnecessary confusion.

As a result of this renaming, some values and arguments in multizone/kubernetes environment changed. You can read below more.

Upgrading with kumactl on Kubernetes

  1. Changes in arguments/flags for kumactl install control-plane

    • --mode accepts now values: standalone, zone and global (remote changed to zone)

    • --tls-kds-remote-client-secret flag was renamed to --tls-kds-zone-client-secret

  2. Service kuma-global-remote-sync changed to kuma-global-zone-sync so after upgrading global control plane you have to manually remote old service. For example:

    kubectl delete -n kuma-system service/kuma-global-remote-sync

    Hint: It's worth to remember that often at this point the IP address/hostname which is used as a KDS address when installing Kuma Zone Control Planes will change. Make sure that you update the address when upgrading the Remote CPs to the newest version.

Upgrading with helm on Kubernetes

Changes in values in Kuma's HELM chart

  • controlPlane.mode accepts now values: standalone, zone and global (remote changed to zone)

  • controlPlane.globalRemoteSyncService was renamed to controlPlane.globalZoneSyncService

  • controlPlane.tls.kdsRemoteClient was renamed to controlPlane.tls.kdsZoneClient

Suggested Upgrade Path on Universal

  1. Zone Control Planes should be started using new environment variables

    • KUMA_MODE accepts now values: standalone, zone and global (remote changed to zone)

      Old:

      KUMA_MODE="remote" [...] kuma-cp run

      New:

      KUMA_MODE="zone" [...] kuma-cp run

    • KUMA_MULTIZONE_REMOTE_ZONE was renamed to KUMA_MULTIZONE_ZONE_NAME

      Old:

      KUMA_MULTIZONE_REMOTE_ZONE="remote-1" [...] kuma-cp run

      New:

      KUMA_MULTIZONE_ZONE_NAME="remote-1" [...] kuma-cp run

    • KUMA_MULTIZONE_REMOTE_GLOBAL_ADDRESS was renamed to KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS

      Old:

      KUMA_MULTIZONE_REMOTE_GLOBAL_ADDRESS="grpcs://localhost:5685" [...] kuma-cp run

      New:

      KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS="grpcs://localhost:5685" [...]  kuma-cp run

    • KUMA_MULTIZONE_REMOTE_KDS_ROOT_CA_FILE was renamed to KUMA_MULTIZONE_ZONE_KDS_ROOT_CA_FILE

      Old:

      KUMA_MULTIZONE_REMOTE_KDS_ROOT_CA_FILE="/rootCa" [...] kuma-cp run

      New:

      KUMA_MULTIZONE_ZONE_KDS_ROOT_CA_FILE="/rootCa" [...] kuma-cp run

    • KUMA_MULTIZONE_REMOTE_KDS_ROOT_CA_FILE was renamed to KUMA_MULTIZONE_ZONE_KDS_ROOT_CA_FILE

      Old:

      KUMA_MULTIZONE_REMOTE_KDS_REFRESH_INTERVAL="9s" [...] kuma-cp run

      New:

      KUMA_MULTIZONE_ZONE_KDS_REFRESH_INTERVAL="9s" [...] kuma-cp run

  2. Dataplane Ingress resource should be replaced with ZoneIngress resource:

    Old:

    type: Dataplane
name: dp-ingress
mesh: default
networking:
  address: <ADDRESS>
  ingress:
    publicAddress: <PUBLIC_ADDRESS>
    publicPort: <PUBLIC_PORT>
  inbound:
  - port: <PORT>
    tags:
      kuma.io/service: ingress

    New:

    type: ZoneIngress
name: zone-ingress
networking:
  address: <ADDRESS>
  port: <PORT>
  advertisedAddress: <PUBLIC_ADDRESS>
  advertisedPort: <PUBLIC_PORT>

    NOTE: ZoneIngress resource is a global scoped resource, it's not bound to a Mesh The old Dataplane resource is still supported but it's considered deprecated and will be removed in the next major version of Kuma

  3. Since ZoneIngress resource is not bound to a Mesh, it requires another token type that is bound to a Zone:

    kumactl generate zone-ingress-token --zone=zone-1 > /tmp/zone-ingress-token

  4. kuma-dp run command should be updated with a new flag --proxy-type=ingress:

    kuma-dp run \
  --proxy-type=ingress \
  --dataplane-token-file=/tmp/zone-ingress-token \
  --dataplane-file=zone-ingress.yaml

Upgrade to 1.1.0

The major change in this release is the migration to XDSv3 for the kuma-cp to envoy data plane proxy communication. The previous XDSv2 is still available and will continue working. All the existing data plane proxies will still use XDSv2 until being restarted. The newly deployed kuma-dp instances will automatically get bootstrapped to XDSv3. In case that needs to be changed, kuma-cp needs to be started with KUMA_BOOTSTRAP_SERVER_API_VERSION=v2.

With Kuma 1.1.0, the kuma-cp will installs default retry and timeout policies for each new created Mesh object. The pre-existing meshes will not automatically get these default policies. If needed, they should be created accordingly.

This version removes the deprecated --dataplane flag in kumactl generate dataplane-token, please consider migrating to use --name instead.

Upgrade to 1.0.0

This release introduces a number of breaking changes. If Kuma is being deployed in production we strongly suggest to backup the current configuration, tear down the whole cluster and zones, and install in a clean setup. However, we enumerate the details of these changes below.

Suggested Upgrade Path on Kubernetes

  • Drop k8s 1.13 support

    Take this into account if you run Kuma on an old Kubernetes version.

  • kumactl merged install ingress into install control-plane

    This change impacts any deployment pipelines that are based on kumactl and are used for multi-zone deployments.

  • Change policies on K8S to scope global

    All the CRDs are now in the global scope, therefore all policies need to be backed up. The relevant CRDs need to be deleted, which will clear all the policies. After the upgrade, you can apply the policies again. We do recommend to keep all the Kuma Control Planes down while doing these operations.

  • Autoconfigure single cert for all services

    Deployment flags for providing TLS certificates in Helm and kumactl have changed, refer to the relevant documentation to verify the new naming.

  • Create default resources for Mesh

    The following default resources will be created upon the first start of Kuma Control Plane - default signing key - default Allow All traffic permission policy allow-all-<mesh name> - Default Allow All traffic route policy allow-all-<mesh name>

    Please verify if this conflicts with your deployment and expected policies.

  • New Multizone deployment flow

    Deploying Multizone clusters is now simplified, please refer to the deployment documentation of the updated procedure.

  • Improved control plane communication security

    Kuma Control Plane exposed ports are reduced, please revise the documentation for detailed list. Consider reinstalling the metrics due to the port changes in Kuma Prometheus SD.

  • Traffic route format

    The format of the TrafficRoute has changed. Please check the documentation and adapt your resources.

Suggested Upgrade Path on Universal

  • Get rid of advertised hostname KUMA_GENERAL_ADVERTISED_HOSTNAME was removed and not needed now.

  • Autoconfigure single cert for all services Deployment flags for providing TLS certificates in Helm and kumactl have changed, refer to the documentation to verify the new naming.

  • Create default resources for Mesh

    The following default resources will be created upon the first start of Kuma Control Plane - default signing key - default Allow All traffic permission policy allow-all-<mesh name> - Default Allow All traffic route policy allow-all-<mesh name>

    Please verify if this conflicts with your deployment and expected policies.

  • New Multizone deployment flow

    Deploying Multizone clusters is now simplified, please refer to the deployment documentation of the updated procedure.

  • Improved control plane communication security

    kuma-dp invocation has changed and now allows for a more flexible usage leveraging automated, template based Dataplane resource creation, customizable data-plane token boundaries and additional CA ceritficate validation for the Kuma Control plane boostrap server. Kuma Control Plane exposed ports are reduced, please revise the documentation for detailed list.

  • Traffic route format

    The format of the TrafficRoute has changed. Please check the documentation and adapt your resources.

Upgrade to 0.7.0

Support for kuma.io/sidecar-injection annotation. On Kubernetes change the namespace resources that host Kuma mesh services with the aforementioned annotation and delete the label.

Prefix the Kuma built-in tags with kuma.io/ as follows: kuma.io/service, kuma.io/protocol, kuma.io/zone.

Suggested Upgrade Path on Kubernetes

Update the applied policy tag selector to include the kuma.io/ prefix. A sample traffic resource follows:

apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
mesh: default
metadata:
  namespace: default
  name: allow-all-traffic
spec:
  sources:
    - match:
        kuma.io/service: '*'
  destinations:
    - match:
        kuma.io/service: '*'

The Kuma Control Plane will update the relevant Dataplane resources accordingly

Suggested Upgrade Path on Universal

Update the applied policy tag selector to include the kuma.io/ prefix. A sample traffic resource follows:

type: TrafficPermission
name: allow-all-traffic
mesh: default
sources:
  - match:
      kuma.io/service: '*'
destinations:
  - match:
      kuma.io/service: '*'

Update the dataplane resources with the new tag format as well. Example:

echo "type: Dataplane
mesh: default
name: redis-1
networking:
  address: 192.168.0.1
  inbound:
  - port: 9000
    servicePort: 6379
    tags:
      kuma.io/service: redis" | kumactl apply -f -

This release changes the way that Distributed and Hybrid Kuma Control planes are deployed. Please refer to the documentation for more details.

Upgrade to 0.6.0

Passive Health Check were removed in favor of Circuit Breaking.

Format of Active Health Check changed from :

apiVersion: kuma.io/v1alpha1
kind: HealthCheck
mesh: default
metadata:
  namespace: default
  name: web-to-backend-check
mesh: default
spec:
  sources:
  - match:
      service: web
  destinations:
  - match:
      service: backend
  conf:
    activeChecks:
      interval: 10s
      timeout: 2s
      unhealthyThreshold: 3
      healthyThreshold: 1
    passiveChecks:
      unhealthyThreshold: 3
      penaltyInterval: 5s

to

apiVersion: kuma.io/v1alpha1
kind: HealthCheck
mesh: default
metadata:
  namespace: default
  name: web-to-backend-check
mesh: default
spec:
  sources:
  - match:
      service: web
  destinations:
  - match:
      service: backend
  conf:
    interval: 10s
    timeout: 2s
    unhealthyThreshold: 3
    healthyThreshold: 1

Suggested Upgrade Path on Kubernetes

In the new Kuma version serivce tag format has been changed. Instead of backend.kuma-demo.svc:5678 service tag will look like this backend_kuma-demo_svc_5678. This is a breaking change and Policies should be updated to be compatible with the new Kuma version.

Please re-install Prometheus via kubectl install metrics and make sure that skipMTLS is set to false or omitted.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: false

Suggested Upgrade Path on Universal

Make sure that skipMTLS is set to true.

type: Mesh
name: default
metrics:
  enabledBackend: prometheus-1
  backends:
  - name: prometheus-1
    type: prometheus
    conf:
      skipMTLS: true

Upgrade to 0.5.0

Suggested Upgrade Path on Kubernetes

Mesh resource format changes

The Mesh resource format in Kubernetes changed from

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabled: true
    ca:
      builtin: {}
  metrics:
    prometheus: {}
  logging:
    backends:
    - name: file-1
      file:
        path: /var/log/access.log
  tracing:
    backends:
    - name: zipkin-1
      zipkin:
        url: http://zipkin.local:9411/api/v1/spans

to

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
  metrics:
    enabledBackend: prom-1
    backends:
    - name: prom-1
      type: prometheus
  logging:
    backends:
    - name: file-1
      type: file
      conf:
        path: /var/log/access.log
  tracing:
    backends:
    - name: zipkin-1
      type: zipkin
      conf:
        url: http://zipkin.local:9411/api/v1/spans

Removing kuma-injector

Kuma 0.5.0 ships with kuma-injector embedded into the kuma-cp, which makes its previously created resources obsolete and potentially can cause problems with the deployments. Before deploying the new version, it is strongly advised to run a cleanup script kuma-0.5.0-k8s-remove_injector_resources.sh.

NOTE: if Kuma was deployed in a namespace other than kuma-system, please run export KUMA_SYSTEM=<othernamespace before running the cleanup script.

Kuma resources ownerReferences

Kuma 0.5.0 introduce webhook for setting ownerReferences to the Kuma resources. If you have some Kuma resources in your k8s cluster, then you can use our script kuma-0.5.0-k8s-set_owner_references.sh in order to properly set ownerReferences .

Suggested Upgrade Path on Universal

Mesh resource format changes

Mesh format on Universal changed from

type: Mesh
name: default
mtls:
  enabled: true
  ca:
    builtin: {}
metrics:
  prometheus: {}
logging:
  backends:
  - name: file-1
    file:
      path: /var/log/access.log
tracing:
  backends:
  - name: zipkin-1
    zipkin:
      url: http://zipkin.local:9411/api/v1/spans

to

type: Mesh
name: default
mtls:
  enabledBackend: ca-1
  backends:
  - name: ca-1
    type: builtin
metrics:
  enabledBackend: prom-1
  backends:
  - name: prom-1
    type: prometheus
logging:
  backends:
  - name: file-1
    type: file
    conf:
      path: /var/log/access.log
tracing:
  backends:
  - name: zipkin-1
    type: zipkin
    conf:
      url: http://zipkin.local:9411/api/v1/spans

Upgrade to 0.4.0

Suggested Upgrade Path on Kubernetes

No additional steps are needed.

Suggested Upgrade Path on Universal

Migrations

Kuma 0.4.0 introduces DB Migrations for Postgres therefore before running the new version of Kuma, run the kuma-cp migration command.

kuma-cp migrate up

Remember to provide config just like in kuma-cp run command. All existing data will be preserved.

New Dataplane Entity format

Kuma 0.4.0 introduces new Dataplane entity format to improve readability as well as add support for scraping metrics of Gateway Dataplanes.

Here is example of migration to the new format.

Dataplane

Old format

type: Dataplane
mesh: default
name: web-01
networking:
  inbound:
  - interface: 192.168.0.1:21011:21012
    tags:
      service: web
  outbound:
  - interface: :3000
    service: backend

New format

type: Dataplane
mesh: default
name: web-01
networking:
  address: 192.168.0.1
  inbound:
  - port: 21011
    servicePort: 21012
    tags:
      service: web
  outbound:
  - port: 3000
    service: backend

Gateway Dataplane

Old format

type: Dataplane
mesh: default
name: kong-01
networking:
  gateway:
    tags:
      service: kong

New format

type: Dataplane
mesh: default
name: kong-01
networking:
  address: 192.168.0.1
  gateway:
    tags:
      service: kong

Although the old format is still supported, it is recommended to migrate since the support for it will be dropped in the next major version of Kuma.

Upgrade to 0.3.1

List of breaking changes

kuma policies:

  • Mesh CRD on Kubernetes is now Cluster-scoped
  • TrafficLog policy is applied differently now: instead of applying all TrafficLog policies that match to a given outbound interface of a Dataplane, only a single the most specific TrafficLog policy is applied

kumactl:

  • a few options in kumactl config control-planes add command have been renamed:
    • --dataplane-token-client-cert has been renamed into --admin-client-cert
    • --dataplane-token-client-key has been renamed into --admin-client-key

Suggested Upgrade Path on Kubernetes

  • Users on Kubernetes will have to re-install Kuma:

    1. Export all Kuma resources 
      kubectl get meshes,trafficpermissions,trafficroutes,trafficlogs,proxytemplates --all-namespaces -oyaml > backup.yaml
    2. Uninstall previous version of Kuma Control Plane 
      # using previous version of `kumactl`

kumactl install control-plane | kubectl delete -f -
    3. Install new version of Kuma Control Plane 
      # using new version of `kumactl`

kumactl install control-plane | kubectl apply -f -
    4. Re-apply Kuma resources back again 
      kubectl apply -f backup.yaml

Suggested Upgrade Path on Universal

  • Those users who used --dataplane-token-client-cert and --dataplane-token-client-key command line options in the past will have to re-run

    kumactl config control-planes add

    this time with

    --admin-client-cert <CERT> --admin-client-cert <KEY> --overwrite

  • all components of Kuma Control Plane - kuma-cp, kuma-dp, envoy - have to be re-deployed