Argo CD is unable to connect with Private GKE

[abhinayguptapeoplegrove]

PROBLEM STATEMENT:

• Unable to add private GKE cluster in Argocd

SUMMARY:

• We have created a public GKE cluster for hosting all ArgoCD components
• We are using this cluster to deploy k8 resources to other multiple private GKE clusters across different GCP projects

STEPS FOLLOWED:

• 1- We created the private GKE cluster and added our IP in the authorised network
• 2- We install the argoCD Cli on our laptop and we did the argocd login
• 3- We are able to access the ArgoCD UI and apps
• 4- To add the private GKE cluster via argocd cli, we executed above command and got the error

ERROR:
We are getting below error while adding the private GKE cluster via argocd CLI

**➜$ argocd cluster add gke_*******_us-******-c_*******
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system" 
INFO[0001] ClusterRole "argocd-manager-role" updated    
INFO[0002] ClusterRoleBinding "argocd-manager-role-binding" updated
FATA[0035] rpc error: code = Unknown desc = Get "https://34.xx.xx.xx/version?timeout=32s": dial tcp 34.xx.xx.xx:443: i/o timeout** 

DEBUG INFORMATION:
Cloud - GCP
Argo version- v2.0.2
GKE version- 1.19.10-gke.1600
Private GKE cluster features enabled:

Control plane global access: TRUE
Control plane authorized networks: TRUE ("Added our public CIDR to it")

P.S: This is an urgent Issue. Please help

[meier-christoph]

You can actually add a cluster without using the cli. A cluster is configured using json in a Secret with some labels to tell argocd that this is the conf of your new cluster. The json contains the token of the service account which you can get from the target cluster. You should already have that done based on the logs you posted. You also need the uri of the api server and the caBundle and I think thats it.

I am currently on my phone and not 100% sure about all the details but we are doing more or less the same thing as you and we don‘t use the cli at all. This part is not so well documented, most ppl should use the cli but its definately possible.

Also, I believe that now there are some NetworkPolicies by default. The trap with NPs is that they work like a whitelist BUT if there are none defined the default is to allow all traffic which is counter intuitive. Since there are now NPs they drop any traffic that is not explicitly defined.

You should try to add a NP that allows argocd to egress to your other clusters. Then exec into the argocd container and test if you can reach the other clusters.

[IamViditAgarwal]

Hi, @meier-christoph
We tried creating the secret in the argocd for adding the cluster , It seems to work. The cluster is added and we can see it the argo dashboard UI . But now we are not able to deploy the application in that cluster . Its giving us the same : Get "https://34.xx.xx.xx/version?timeout=32s": dial tcp 34.xx.xx.xx:443: i/o timeout**
Could you share the networkpolicy ? Not able to figure-out what should be the ideal policy to be created . That will be really helpful

Thanks & Regards

[cheskayang]

^ run into the same issue, were you able to resolve it?

[meier-christoph]

Try something like this :

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-to-external-cluster
  namespace: argocd
spec:
  podSelector: {}
  egress:
    - to:
        - ipBlock:
            cidr: 34.xxx.xxx.xxx/32
      ports:
        - port: 443

This allows all pods in the argocd namespace to connect with your IP on port 443. For production you should change the podSelector to only select the actual pod but for testing this should be ok.

There is a great tool to create NPs if you are not familiar with the syntax
https://editor.cilium.io/

Next, you should just exec into the pod and try to ping the target server. NPs are just firewall rules managed by k8s, you should treat this like any other networking issue. It's also possible that you need to add some networking rules in GCP to make it work.

If I remember correctly there is kubectl in the argocd docker image, if you exec into the container you can manually create the ~/.kube/config file for your target cluster and see if it works.

[abhinayguptapeoplegrove]

Hey @meier-christoph . Thanks for your suggestion.

We have created the same NP in kubernetes cluster as you mentioned but it didn't work, even after adding firewalls given below, it didn't work

ArgoCD cluster project firewall rule:

• Type- egress, protocols and port- all
• Applicable On- for all Argcd Cluster nodes (ArgoCD server is running on one of the nodes)
• Destination IP- 34.xxx.xxx.xxx/32 (This is public endpoint of a private GKE nodes), also tested with private end point (10.xx.xx.xx/32) as destination ip in firewall rule

Target private GKE cluster project firewall rule:

• Type- ingress, protocols and port- from all
• Applicable On- For all private GKE cluster nodes

For verification
From argocd project, telnet for master ip( target private GKE project), port 443 is not working

[meier-christoph]

Well the good news is that since telnet is not working you know for sure that this is a network issue and not an ArgoCD issue.

I don't really know much about GCP but at this point its safe to assume that there are some firewalls blocking traffic that need to be configured somewhere.

Also, if you are not using NPs (e.g. older versions of argocd) I would recommend removing the ones you created and use the default behavior which is allow all (or you create the necessary rules for all pods that argocd uses).

[abhinayguptapeoplegrove]

We checked the argocd version and found below details below, it looks latest version

argocd: v1.8.7+eb3d1fb.dirty
  BuildDate: 2021-03-07T19:33:57Z
  GitCommit: eb3d1fb84b9b77cdffd70b14c4f949f1c64a9416
  GitTreeState: dirty
  GoVersion: go1.16
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.0.4+0842d44
  BuildDate: 
  GitCommit: 
  GitTreeState: 
  GoVersion: 
  Compiler: 
  Platform: 
  Ksonnet Version: 
  Kustomize Version: 
  Helm Version: 
  Kubectl Version: 
  Jsonnet Version:

If possible can you please share some docs for reference for this type of Use-case

[meier-christoph]

NPs were added in v2.0.2 #6156 you should be fine with the latest version

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-to-external-cluster
  namespace: argocd
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 34.0.0.0/8
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8
    - to:
        - podSelector: {}

you can also try this which is basically allowing any traffic but again it will probably not do much since the real issue is probably a missing configuration in GCP and I won't be able to help with that since I never used it.

[davinerd]

I'm struggling with this as well.

I've downgraded to ArgoCD 2.0.0 thinking it would something NP related, but it actually isn't.

Based on Google official doc, when creating manually the argocd secret, the server section must be the private endpoint IP (which can be obtained with a gcloud container clusters describe).

Still, the status of the cluster in the ArgoCD UI is "Unknown", which is not what we expect.

Firewall rules on GCP allow TCP 443 communication between nodes and master plane. Workloads are deployed correctly, thus I'd exclude a firewall issue on GCP side.

Could we have more support on this issue?
It's disappointing to see ArgoCD not taking into serious consideration this issue, taking into account that private GKE clusters should be the norm and thus ArgoCD should work out of the box, or at least provide documentation on how to make it work.

Edit: adding a firewall rule in GCP with allow all protocols for all machines in the network coming from 0.0.0.0/0 still doesn't solve the issue.

[mdtoro-wyn]

The egress (NAT) IP of where ever ARGO is running (Cluster A), must be allowed in control plane authorized networks (Cluster B). That solved our issue.

[cloudgk]

This fixed the issue, thank you very much!!

[szihai]

@mdtoro-wyn, by "egress IP", do you mean the Ingress or LB ip, depends on how Argo exposes the service? Mine was, but it didn't work.

[mdtoro-wyn]

Cloud NAT IP. It is located in network services/ Cloud NAT/ NAT IP addresses (Cluster A where Argo is installed) and then add that IP in control plane authorized networks (Cluster B). In our case we added the IP range in this format: xx.xxx.xxx.xxx/32

[igitcode]

Adding cloud NAT IP to authorized networks of my target cluster fixed this issue for me as well.

[bvboca]

I found the same issue on on-prem kubernetes 1.19, when connecting in the same cluster from slave1 to master. Applying the NP rule above is not resoving the issue. Does it mean when in the same cluster sa.yaml above will do the job of "argocd cluster add kubernetes-admin@kubernetes"?

INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
FATA[0030] rpc error: code = Unknown desc = Get "https://192.168.X.121:6443/version?timeout=32s": dial tcp 192.168.X.121:6443: i/o timeout

[meier-christoph]

Hi, you only need the argocd-manager when connecting to a different cluster, in order to deploy „in-cluster“ argocd will use the service account it is running with using the auto mounted tokens it has in the container i.e you don‘t have anything to do.

[bvboca]

Got it! Thanks!

[szihai]

Having the same problem on GKE. Already we have created authorized network for the clusters. All the node ips and API server IPs are in that range.

[szihai]

Just tried nc command from network utility pod on both clusters. They can connect to each other's api server.

[szihai]

Also when I run argco cluster add, it went smoothly without any connection error. However on the receiving side, nothing was created. And cluster status is unknown.

[kwsorensen]

Well I started looking into this after having used ArgoCD in multiple different companies. Looks like GKE Private clusters do not allow network peering for the control plane. see https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept#the_control_plane_in_private_clusters

Every GKE cluster has a Kubernetes API server that is managed by the control plane (master).

The control plane runs on a virtual machine (VM) that is in a VPC network in a Google-owned project. A regional cluster has multiple control planes, each of which runs on its own VM.

In private clusters, the control plane's VPC network is connected to your cluster's VPC network with VPC Network Peering. Your VPC network contains the cluster nodes, and the Google-owned Google Cloud VPC network contains your cluster's control plane.

Traffic between nodes and the control plane is routed entirely using internal IP addresses. If you use VPC Network Peering to connect your cluster's VPC network to a third network, the third network cannot reach resources in the control plane's VPC network. This is because VPC Network Peering only supports communication between directly peered networks, and the third network cannot be peered with the control plane network. For more information, see VPC Network Peering restrictions.

[kwsorensen]

I also believe ArgoCD needs to access the Kubernetes API Server to be able to create/monitor Kubernetes resources.

It them seems like a logcal step that Private GKE clusters cannot be managed by an external ArgoCD.

Does ArgoCD have a solution for this scenario in the pipeline?

[MachiAngel]

Having the same problem and i figure out how to set

here is my scene
i have two different project
each one have gke (GKE-A , GKE-B)
install argocd in GKE-A
as same problem , argocd cluster add will time out to GKE-B
after set GKE-A nat ip to GKE-B "Control plane authorized networks"
it's work

just go VPC network -> External IP Address and find ur GKE-A ip

[elebioda]

I think a connect gateway can solve this private cluster problem. I am using it rite now, but there is a problem where if the agent crashes argocd no longer show's the resources and I have to invalidate the cluster cache. It makes sense, but I don't like that I have to do it manually and argocd doesn't just see it is no longer watching the resources and just does the kubectl watch again.

[adle29]

I am not sure what IP you used, there are several VM instances to take the external IP. I added them all and still could not link the private cluster. Also if they are ephemeral external IPs I believe argocd will have issues later.

I thought with the Control plane authorized networks this would work, but still cannot seem to make it work.

[ls0f]

#9496 [manage clusters via proxy] may be a solution for private cluster

[shanproofpoint]

when creating a private gke cluster did you happen to specify master authorized networks? make sure the argocd node/cloud nat is allowed to enter the newly created target gke cluster

[zhang-xuebin]

If anyone is still looking for a solution, https://github.com/zhang-xuebin/argo-helm/tree/host-on-gke is an example. You can use ArgoCD to manage Private GKE clusters. VPC-peering, master Authorized Networking, are NOT needed.

See also: https://cloud.google.com/blog/products/containers-kubernetes/connect-gateway-with-argocd

[defyjoy]

Hi @zhang-xuebin I tried this approach but something is missing . I would be glad if you could help me out .

[zhang-xuebin]

What is missing? Can you share some more details?

[defyjoy]

@zhang-xuebin

Sorry for the delayed response
This is the cluster fleet

But when I register the service gke - It gives me this error -

The module configuration is here for the service cluster -

module "gke_dev" {

  source = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"

  version            = "~> 31.0"
  kubernetes_version = "1.28.9"

  name = "${var.name}-${var.environment}-gke-cluster"

  deletion_protection = false

  gateway_api_channel = "CHANNEL_STANDARD"

  # Required variables
  project_id = var.project_id
  region     = var.region

  fleet_project_grant_service_agent = true
  fleet_project                     = var.fleet_project

  # network_project_id = "${var.name}-management"
  network           = var.network_name
  subnetwork        = "gke-cluster-subnet-1"
  ip_range_pods     = "gke-cluster-pod-subnet-1"
  ip_range_services = "gke-cluster-service-subnet-1"

  service_account_name = "${var.name}-${var.environment}-gke"

  enable_l4_ilb_subsetting = true

  # Other variables
  http_load_balancing        = true
  network_policy             = true
  horizontal_pod_autoscaling = true
  gce_pd_csi_driver          = true

  filestore_csi_driver = true

  stack_type = "IPV4"

  enable_private_endpoint  = true
  enable_private_nodes     = true
  remove_default_node_pool = true

  enable_cost_allocation          = true
  enable_shielded_nodes           = true
  enable_vertical_pod_autoscaling = true

  grant_registry_access = true
  registry_project_ids  = [var.project_id]

  master_global_access_enabled = true
  enable_intranode_visibility  = true
  gke_backup_agent_config      = true

  master_ipv4_cidr_block = var.master_ipv4_cidr_block

  master_authorized_networks = var.master_authorized_networks

  # cluster_resource_labels = local.default_labels
  node_metadata = "GKE_METADATA_SERVER"

  node_pools = [
    {
      name         = "default-pool"
      machine_type = "e2-standard-2"

      min_count          = 1
      max_count          = 100
      initial_node_count = 1

      local_ssd_count = 0

      spot         = true
      disk_size_gb = 100
      disk_type    = "pd-standard"
      image_type   = "UBUNTU_CONTAINERD" # "COS_CONTAINERD"
      enable_gcfs  = false
      enable_gvnic = false
      auto_repair  = true
      auto_upgrade = true
      # service_account    = "project-service-account@<PROJECT ID>.iam.gserviceaccount.com"
      preemptible = false

      # node_config = {
      #   dynamic "workload_metadata_config" {
      #     node_metadata = "GKE_METADATA_SERVER"
      #   }
      # }
    }
  ]

  node_pools_oauth_scopes = {
    all = [
      "https://www.googleapis.com/auth/cloud-platform"
      # "https://www.googleapis.com/auth/logging.write",
      # "https://www.googleapis.com/auth/monitoring",
    ]
  }

  node_pools_labels = {
    all = {}

    default-pool = var.gke_node_pool_default_labels
    # longhorn     = merge(local.default_labels, local.longhorn_labels)
  }


  node_pools_tags = {
    all = []

    default-node-pool = [
      "default-node-pool",
    ]
  }
}

[didlawowo]

Hi,
i have similar issue
i have one cluster classic with argo cd and another private
i have the same issue about firewall,
after adding cloud nat with manual ip, vpc peering, and firewall rules, and add cloud nat into authorized

i always have the problem, i'm using argo 2.10,

maybe something that a i don't understand any help would be appreciated