send notifications from the argocd-notification-controller pod to Splunk via webhook notifications

[prithvi-bit]

Hi Argo CD experts,

We are currently using Argo CD version v2.11.4+e1284e1 and have enabled the ArgoCD notification controller component. Our goal is to send notifications from the argocd-notification-controller pod to Splunk via webhook notifications, triggered when a new app is deployed or an app sync succeeds. However, despite configuring everything correctly to trigger events on sync success, no events are being triggered from the argocd-notification-controller. There are no errors logged in the controller pod, and no events appear in Splunk, making it very difficult to troubleshoot the issue since there are no visible errors or logs to investigate.

Please find my code snippet and help me where i have made wrong.

` service.webhook.splunk: |
url: https://$(splunk_url):8088/services/collector
headers:
- name: Content-Type
value: application/json
- name: Authorization
value: Splunk $SPLUNK_TOKEN

splunksubscriptions: |
- recipients:
- splunk # service name, not service type
triggers:
- splunk-on-sync-status

template.splunk-app-sync-status: |
webhook:
notif:
method: POST
body: |
{ "records":
[
{
"source": "ArgoCD",
"additional_info": "{ "event-type": "argocd-app-status", "event-source": "ArgoCD", "event-time-sent": "{{{{ (call .time.Now).Format "2006-01-02T15:04:05" }}}}", "argocd-url": "{{{{ .context.argocdUrl }}}}", "startDate": "{{{{ .app.status.operationState.startedAt }}}}", "finishDate": "{{{{ .app.status.operationState.finishedAt }}}}", "application": "{{{{ .app.metadata.name }}}}", "sync-status": "{{{{ .app.status.sync.status }}}}", "health-status": "{{{{ .app.status.health.status }}}}", "repo-url": "{{{{ .app.spec.source.repoURL | call .repo.RepoURLToHTTPS }}}}", "revision": "{{{{ .app.status.sync.revision }}}}", "author": "{{{{ (call .repo.GetCommitMetadata .app.status.sync.revision).Author }}}}", "destination-cluster": "{{{{ .app.spec.destination.server }}}}", "u_stack": "{{{{ .app.spec.destination.name }}}}", "destination-namespace": "{{{{ .app.spec.destination.namespace }}}}", "reconciled-at": "{{{{ (call .time.Parse .app.status.reconciledAt).Format "2006-01-02T15:04:05" }}}}" }"
}
]
}

trigger.splunk-on-sync-status: |
- description: Application sync status update
send:
- splunk-app-sync-status
when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'`

Any help would be much appreciated.

Thanks,
Prithvi