Configure CMP via sidecar: error="Manifest generation error: plugin sidecar failed. Couldn't find cmp-server plugin supporting repository .env/dev/workloads/my-app"

[azamora-zsc]

Hi,
I'm taking this other discussion as a reference: #8216

In my case, I also had my plugin working using the ConfigMap option. Now I need to change this and configure my CMP via a sidecar like this.

The plugin is in this case a tool developed by us that needs to access the code in the git repository to work properly. I have done several tests trying to make this sidecar approach work but I keep getting the following error:

time="2022-05-12T13:06:56Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error: plugin sidecar failed. Couldn't find cmp-server plugin supporting repository .env/dev/workloads/my-app" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-05-12T13:07:56Z" grpc.service=repository.RepoServerService grpc.start_time="2022-05-12T13:06:56Z" grpc.time_ms=1.087 span.kind=server system=grpc

Details

Versions:

• ArgoCD Helm chart version: 4.5.10
• ArgoCD version: 2.3.3

Plugin configuration. As you can see, I've used the same trick of "touching" a file in /tmp to see if the plugin is called, and returning always a non-error code as mentioned in the documentation.

apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
  name: configurator-plugin
spec:
  version: v1.0.1
  generate:
    command: [sh, -c, 'configurator generate --all -o yaml']
  discover:
    find:
      command: ["touch", "/tmp/plugin-discover-has-run"]
  allowConcurrency: true
  lockRepo: false

The file in /tmp it's created and everything looks good to me. Here some relevant log entries in the sidecar container (cmp) logs:

time="2022-05-12T12:39:56Z" level=info msg="touch /tmp/plugin-discover-has-run" dir=/tmp/_argocd-repo/6c8a95fe-5267-41f7-9bea-719063be6cda/env/dev/workloads/my-app execID=21591
time="2022-05-12T12:39:56Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.request.deadline="2022-05-12T12:40:55Z" grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-05-12T12:39:56Z" grpc.time_ms=0.683 span.kind=server system=grpc 

Here the sidecar definition added repo server deployment:

repoServer:
  extraContainers:
    - name: cmp
      command: [/var/run/argocd/argocd-cmp-server] # Entrypoint should be Argo CD lightweight CMP server i.e. argocd-cmp-server
      image: my-registry.url/configurator-plugin:v1.0
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
      - mountPath: /var/run/argocd
        name: var-files
      - mountPath: /home/argocd/cmp-server/plugins
        name: plugins
      - mountPath: /tmp
        name: tmp-dir

And finally an application using the plugin:

project: config-management
source:
  repoURL: '[email protected]:myorg/repo-config.git'
  path: env/dev/workloads/my-app
  targetRevision: HEAD
  plugin: {}
destination:
  server: 'https://XXXXXXXX.xxx.eu-central-1.eks.amazonaws.com'
  namespace: monitoring
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  syncOptions:
    - Validate=false
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m

I'm following the documentation to the letter and I can't find a solution for this situation. Something that is calling my attention is that all the examples (I couldn't find many) use an echo instruction to generate the manifests, so they are almost useless in real life, where the plugin binary will probably depend on the source code in the cloned repository.

Can anybody help with this issue? Maybe we can use this discussion to get a more complete example that other people can use in case of a need to use a sidecar for configuring a CMP. (I read that this could be the only option in the future).

Thanks in advance.

Last-minute update:
If I remove the /tmp volume mount as I read in the documentation, then something similar to this happens.

[crenshaw-dev]

The docs are incomplete in the stable version. The latest docs include this line:

The find command should return a non-error exit code and produce output to stdout when the application source type is supported.

The find command has to have non-empty output. echo hi should work.

[crenshaw-dev]

@azamora-zsc are you mounting the same temp dir volume in both repo-server and the sidecar? If so, I'd create a new temp volume for the sidecar. We specifically want to avoid sharing the temp volume between repo-server and plugin sidecars, to secure against path traversal attacks.

[btrepp]

I'm getting similar behaviour using

discover:
    fileName: "kubepassgen.toml"

For filename, does the file need to be in the root of the git repository? or the root of the app.
I think

discover:
    fileName: "$ARGOCD_APP_SOURCE_PATH/kubepassgen.toml"

didn't seem to work either.

File of course exists :). I might have to switch to using some find command, as i'm unsure what fileName actually needs or where it looks, but for my purposes if that file is there, my plugin is good to go.

  discover:
    find:
      command: [sh, -c, find $ARGOCD_APP_SOURCE_PATH -name kubepassgen.toml]

Doesn't work for me either.

So I am a bit stumped to make find work. I could just true true and not search for files,

[crenshaw-dev]

does the file need to be in the root of the git repository? or the root of the app.

It needs to be in the root of the git repo.

discover:
    fileName: "$ARGOCD_APP_SOURCE_PATH/kubepassgen.toml"

I don't think env vars are evaluated before running the glob, so I wouldn't expect this to work.

  discover:
    find:
      command: [sh, -c, find $ARGOCD_APP_SOURCE_PATH -name kubepassgen.toml]

I would expect this to work. Any interesting logs in your CMP sidecar?

[btrepp]

Thanks for the confirmation and tips. There was no logs in the sidecar.
I restarted redis and argocd-server and it started working.

I suspect there is some caching in definitions?. Eg my sidecar in dev I do restart often, but it does seem argocd holds onto a few things.
I'm not sure what best to restart when tweaking, but at least it's a hint for everyone else, and crenshaws tips should help future people :)

[azamora-zsc]

Hi @crenshaw-dev.
Thank you very much for responding so quickly. I have tested using a different command that matches the expectations you mentioned, in this case:
$ cat file.libsonnet (this file has to be in the application root, ensuring then that the repository is supported by the plugin)
This did the trick and everything started working.

About your second point related to the /tmp volume. Yes, I had it configured and I kept it for the test above. Once everything was in green in ArgoCD, I removed the volume mount and then I started having this errors in the sidecar:

time="2022-05-13T08:31:04Z" level=info msg="cat file.libsonnet" dir=/tmp/_argocd-repo/dbbad7c4-88c6-4937-9882-dcb938b5474d/env/dev/workloads/my-app execID=61f45
time="2022-05-13T08:31:04Z" level=error msg="finished unary call with code Unknown" error="fork/exec /bin/cat: no such file or directory" grpc.code=Unknown grpc.method=MatchRepository grpc.request.deadline="2022-05-13T08:32:02Z" grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-05-13T08:31:04Z" grpc.time_ms=43.858 span.kind=server system=grpc

and this in the repo server container:

time="2022-05-13T08:31:04Z" level=error msg="finished unary call with code Unknown" error="plugin sidecar failed. Couldn't find cmp-server plugin supporting repository .env/dev/workloads/my-app" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-05-13T08:32:03Z" grpc.service=repository.RepoServerService grpc.start_time="2022-05-13T08:31:03Z" grpc.time_ms=977.836 span.kind=server system=grpc

Looks like I need to restore the volume. You mentioned using a different volume for the sidecar. Is this mandatory or using the file system inside the container should be enough?

[crenshaw-dev]

Oh, I'm a fool. I thought you'd need to remove the shared directory in 2.3, but that's not the case until 2.4. That's why when Raymond upgraded to the 2.4 rc, the error went away. Please ignore me. 😆

[azamora-zsc]

🤣 don't worry... I sort of knew it was like you tell me now, I just wanted confirmation from an expert. In the end, you have solved my problem and we are letting a good thread for people in the same situation in the future, so thank you very much again.

[crenshaw-dev]

Haha you bet, thanks for your patience!

[ronaldour]

I was having the same error and it was very annoying to troubleshoot. In case it helps people, I noticed that sometimes argocd caches the result of the generate or discovery commands and even though I was changing the configuration the result was the same, so I had to restart redis each time. It took me a long time to figure out so I hope this helps someone else.