Could not load TLS keypair: tls: failed to find any PEM data in certificate input

[Chili-Man]

Describe the bug
The Vault agent injector pod had restarted, for reasons that we don't know yet, and the new replacement that came up immediately began spamming the logs with:

{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002300Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002353Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002409Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002454Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002514Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002530Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002567Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002581Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002623Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002669Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002717Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002756Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002815Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002859Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.002892Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003173Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003214Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003227Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003288Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003339Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003401Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003441Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003451Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003514Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003557Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003604Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003643Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003704Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003756Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003791Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003838Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003882Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003928Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003969Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.003979Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004022Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004068Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004117Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004188Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004233Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004280Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004319Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004329Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004356Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004408Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004465Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004517Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004570Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004616Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004670Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004715Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004768Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004807Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004834Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004886Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004929Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004987Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005027Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005037Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005082Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005128Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005179Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005218Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005237Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005308Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005360Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005377Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005424Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005473Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005503Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005566Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005629Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005660Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005694Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005734Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005756Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005804Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005868Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005932Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.005968Z"}
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.006031Z"}

This caused pods to fail to come up because vault agent injector was unresponsive due to the error above. We were able to fix this issue by forcibly restarting the vault agent injector pod.

To Reproduce
I'm not sure how to reproduce; this is the first time we're seeing this issue.

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "12"
    meta.helm.sh/release-name: vault
    meta.helm.sh/release-namespace: vault
  creationTimestamp: "2021-03-24T21:17:41Z"
  generation: 12
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vault-agent-injector
    component: webhook
  name: vault-agent-injector
  namespace: vault
  resourceVersion: "1101913262"
  uid: 1746bdfa-61f3-4092-9196-f78e394e9686
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: vault
      app.kubernetes.io/name: vault-agent-injector
      component: webhook
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        ad.datadoghq.com/sidecar-injector.check_names: |
          ["openmetrics"]
        ad.datadoghq.com/sidecar-injector.init_configs: |
          [{}]
        ad.datadoghq.com/sidecar-injector.instances: |
          [
            {
              "prometheus_url": "https://%%host%%:8080/metrics",
              "namespace": "vault-injector",
              "metrics": [ "*" ],
              "tls_verify": false,
              "tls_ignore_warning": true
            }
          ]
        core.noteable.io/entity: noteable
        kubectl.kubernetes.io/restartedAt: "2023-06-15T13:39:29-07:00"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: vault
        app.kubernetes.io/name: vault-agent-injector
        component: webhook
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: core.noteable.io/node-role
                    operator: In
                    values:
                      - default
      containers:
        - args:
            - agent-inject
            - 2>&1
          env:
            - name: AGENT_INJECT_LISTEN
              value: :8080
            - name: AGENT_INJECT_LOG_LEVEL
              value: info
            - name: AGENT_INJECT_VAULT_ADDR
              value: https://vault.vault.svc:8200
            - name: AGENT_INJECT_VAULT_AUTH_PATH
              value: auth/kubernetes
            - name: AGENT_INJECT_VAULT_IMAGE
              value: hashicorp/vault:1.12.5
            - name: AGENT_INJECT_TLS_CERT_FILE
              value: /etc/webhook/certs/tls.crt
            - name: AGENT_INJECT_TLS_KEY_FILE
              value: /etc/webhook/certs/tls.key
            - name: AGENT_INJECT_LOG_FORMAT
              value: json
            - name: AGENT_INJECT_REVOKE_ON_SHUTDOWN
              value: "false"
            - name: AGENT_INJECT_TELEMETRY_PATH
              value: /metrics
            - name: AGENT_INJECT_CPU_REQUEST
              value: 30m
            - name: AGENT_INJECT_CPU_LIMIT
              value: 500m
            - name: AGENT_INJECT_MEM_REQUEST
              value: 32M
            - name: AGENT_INJECT_MEM_LIMIT
              value: 32M
            - name: AGENT_INJECT_DEFAULT_TEMPLATE
              value: map
            - name: AGENT_INJECT_TEMPLATE_CONFIG_EXIT_ON_RETRY_FAILURE
              value: "true"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
          image: hashicorp/vault-k8s:1.2.1
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 2
            httpGet:
              path: /health/ready
              port: 8080
              scheme: HTTPS
            initialDelaySeconds: 5
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 5
          name: sidecar-injector
          readinessProbe:
            failureThreshold: 2
            httpGet:
              path: /health/ready
              port: 8080
              scheme: HTTPS
            initialDelaySeconds: 5
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits:
              memory: 128Mi
            requests:
              cpu: 200m
              memory: 128Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          startupProbe:
            failureThreshold: 12
            httpGet:
              path: /health/ready
              port: 8080
              scheme: HTTPS
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 5
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/webhook/certs
              name: webhook-certs
              readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 100
      serviceAccount: vault-agent-injector
      serviceAccountName: vault-agent-injector
      terminationGracePeriodSeconds: 30
      volumes:
        - name: webhook-certs
          secret:
            defaultMode: 420
            secretName: vault-injector-tls
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: "2023-06-15T19:19:05Z"
      lastUpdateTime: "2023-06-15T19:19:05Z"
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
    - lastTransitionTime: "2021-03-24T21:17:41Z"
      lastUpdateTime: "2023-06-15T20:39:39Z"
      message: ReplicaSet "vault-agent-injector-767fc9b8c8" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
  observedGeneration: 12
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Other useful info to include: kubectl describe deployment <app> and kubectl describe replicaset <app> output.

Expected behavior
When the vault agent injector restarts, and fails due to
{"@level":"warn","@message":"Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...","@module":"handler.certwatcher","@timestamp":"2023-06-15T20:38:43.004280Z"}

It shouldn't infinitely spam that message as fast as possible. If it fails to load the TLS keypair, it might make sense to have it fail its health checks so that it doesn't keep running.

Environment

• Kubernetes version: EKS 1.26
• vault-k8s version: 1.2.1

Additional context
We're currently looking for additional information right now, we'll post more as we find out more.
We do make use of cert-manager to provide the tls certs to vault-agent-injector

Before the previously working vault agent injector had exited with exit code 2 , before restarting into this bad state.

[lboynton]

Just encountered this too, we had to kill the pod manually so a new one could start in its place.