Syncing some resources before presync hooks run.

[endophage]

Maybe I’m missing something in the docs, I have presync hooks that update a CDN and run database migrations. At least for the DB migrations, this aligns with what is suggested in the docs. However, between these 2 hooks, I need a service account (deploying in Amazon’s EKS where the service account provides access to an S3 bucket we front with Cloudfront), and a secret populated from AWS Secrets Manager via aws-secret-operator.

The first time I go to sync that app on a new cluster, the hooks fail to run because Argo doesn’t create the service account or the secret until the Sync phase. So, I have to selectively sync these resources, then run the full sync with everything else. It’s annoying and seems like a really obvious use case.

Is there some way to have specified resources sync during/at the start of the PreSync phase so they’re available for the hooks? It seems like hooks are only meant to be jobs/pods so I haven’t tried simply annotating the resources with the presync hook. Even if that works, what do I do for the delete policy? I don’t want these to ever be deleted, only updated.

[ishitasequeira]

@endophage You can try using sync waves to create service accounts in wave 1 and then other resources in later waves.

[endophage]

Sync waves are only ordered within the phase. The PreSync phase runs before the Sync Phase so even if I set a sync wave of "-1" on my service accounts and secrets, that only makes them run before other things in the Sync phase, which is still after the PreSync phase.

[ohdearaudrey]

Presync can only contain a hook. To do this you would want to put everything in a sync wave so when your sync starts even though you aren’t “presync” anymore your CDN will update on syncwave 0, syncwave 1 will make your service account. Syncwave 2 will run db migrations. Even though you are in the sync phase you can control the order in which the steps happen.

[endophage]

The order of my tasks isn't the problem. The CDN update and DB migration are entirely independent. It's solely deploying the ServiceAccount and Secret resources before the PreSync tasks run that's the issue.

[endophage]

I've been testing moving the hooks to run as "Sync" hooks rather than "PreSync." I'm seeing the sync continue to update the pods even when the "Sync" hooks fails. This is hugely problematic as it would result in an updated app being deployed even though a database migration its relying on may have failed. While the health checks should prevent the updated app from being brought live, and the pods of the previous version would remain, I don't know that that's going to be guaranteed in all cases and shouldn't even be happening given the failure of the hooks.

[ohdearaudrey]

In a syncwave, if a previous job fails the sync will not continue. So if wave 2 fails wave 3 will never happen but Argo depends on the exit codes of the job to know if something failed. So doing a curl will not return a fail if the request was valid but if the api request you sent has a failure state argo wouldnt know that etc.

[endophage]

That doesn't appear to be whats happening. I'm not only seeing deployments roll-over if the Sync hook fails, I'm seeing them deploy before the Sync hook even finishes. The hooks can take a few minutes to run so it's very obvious that the deployment is happening, and the new pod coming up healthy, before the Hook finishes. The pod takes maybe 60-90s to update while the hook can take 5-6 minutes.

[endophage]

I'm wondering if there's a bug in ArgoCD whereby if any resources have a sync wave set, all other resources are assumed to also have a sync wave set and in its absence are getting Go's zero value.

In my case I have sync waves of "-1" set on the configmaps, secrets, and service accounts. Those do clearly deploy first, then everything else seems to update at the same time as the Sync hook runs.

Setting an explicit sync-wave, I tested with "10" on the Deployment resources now causes them to correctly wait for the Sync hooks to finish.