Running ArgoCD UI in AWS VPC and accessing via Traefik with no SSL Cert; is insecure mode safe for production use?

[k-lombard]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

The ArgoCD UI that I am hosting internally in a private subnet of my AWS VPC on my EKS Cluster does not have a valid SSL Cert and I could not get one working with cert-manager. Is running the argocd-server in insecure mode safe for production use?

Alternatively, looking at my k8s templates, what needs to be fixed in order to get the certificate to be working and validated?

To Reproduce
certificate.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: argocd-cert
  namespace: argocd
spec:
  secretName: argocd-server-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: argocd.staging.redacted.com
  dnsNames:
    - argocd.staging.redacted.com

cluster-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: redacted
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - dns01:
          route53:
            region: us-east-2
            hostedZoneID: redacted
            accessKeyID: redacted
            secretAccessKeySecretRef:
              name: route53-secret
              key: secret-access-key

argocd-cm.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.staging.redacted.com
  server.insecure: 'true'
  server.x-forwarded-proto: 'true'
  server.x-forwarded-host: 'true'
  server.x-forwarded-prefix: 'true'

argocd-ingress.yaml:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: argocd-server
  namespace: argocd
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`argocd.staging.redacted.com`)
      priority: 10
      services:
        - name: argocd-server
          port: 80
    - kind: Rule
      match: Host(`argocd.staging.redacted.com`) && Headers(`Content-Type`, `application/grpc`)
      priority: 11
      services:
        - name: argocd-server
          port: 80
          scheme: h2c
  tls:
    secretName: argocd-server-tls

argocd-network-policy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-server-network-policy
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: argocd-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: traefik
      ports:
        - protocol: TCP
          port: 8080

argocd-server-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: argocd-server
  namespace: argocd
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server

Expected behavior

Expected behavior is that https works properly for the ArgoCD UI with a valid certificate.

Screenshots

Version

argocd: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T15:33:41Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.23.0
  Compiler: gc
  Platform: darwin/arm64

Logs

Paste any relevant application logs here.

[blakepettersson]

@k-lombard why don't you configure an ALB in front of the Argo CD UI/API? Using aws-load-balancer-controller with ACM?

If you still want to use Lets Encrypt / cert-manager / Traefik I'd take a look at the cert-manager docs and see where the provisioning of the certificate went wrong (e.g check kubectl get cert, kubectl get order etc etc..)

[k-lombard]

@blakepettersson I tried internal NLB (detailed here #19894) and ALB, which caused me to run into similar issues. I was only able to even get the UI to work (without a cert) using Traefik. I'm just trying to debug to get HTTPS working.