A single Argo CD instance can deploy to many Kubernetes clusters from one Git repository. In this tutorial you will register two remote clusters in an existing Argo CD installation, generate one Application per cluster with an ApplicationSet, restrict a team to its own project with RBAC, control the order of resources during a sync with waves and hooks, and send Slack messages when a sync fails.
Prerequisites
To follow this tutorial, you will need:
- Argo CD 2.8 or newer (3.x works the same) running in the
argocdnamespace of a management cluster, and theargocdCLI logged in to it with an admin account (argocd login your_argocd_host). - Two additional Kubernetes clusters to deploy to, for example two clusters on CubePath VPS instances, with their contexts in your local kubeconfig. This guide calls them
prod-euandprod-us. Their API servers must be reachable from the pods of the management cluster. kubectlconfigured for the management cluster.- For Step 6: a Slack workspace where you can create an app and get a bot token.
Every Kubernetes manifest in this guide is applied to the management cluster, where Argo CD runs.
Step 1 - Registering the target clusters
argocd cluster add reads a context from your kubeconfig, creates a ServiceAccount called argocd-manager in the kube-system namespace of that cluster, and stores its token in a Secret inside Argo CD. Labels on that Secret let ApplicationSets select clusters later, so add an environment label now. List your contexts first:
kubectl config get-contexts -o name
Register both clusters. Replace the context names with yours:
argocd cluster add prod-eu-context --name prod-eu --label env=prod
argocd cluster add prod-us-context --name prod-us --label env=prod
The command warns that it will create a ServiceAccount with cluster-admin level access and asks for confirmation. Answer y. Then list the clusters:
argocd cluster list
SERVER NAME VERSION STATUS MESSAGE PROJECT
https://203.0.113.20:6443 prod-eu Unknown Cluster has no applications and is not being monitored.
https://203.0.113.30:6443 prod-us Unknown Cluster has no applications and is not being monitored.
https://kubernetes.default.svc in-cluster 1.33 Successful
Unknown is expected until an application targets the cluster. Confirm the labels were stored on the cluster Secrets:
kubectl -n argocd get secrets -l argocd.argoproj.io/secret-type=cluster --show-labels
Step 2 - Creating a project for the team
An AppProject limits which repositories a group of applications can deploy from, where they can deploy to and which cluster-scoped resources they may create. Create a project for a web team that can only deploy into the guestbook namespace:
nano project-team-web.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-web
namespace: argocd
spec:
description: Web team applications
sourceRepos:
- https://github.com/argoproj/argocd-example-apps.git
destinations:
- server: "*"
namespace: guestbook
clusterResourceWhitelist:
- group: ""
kind: Namespace
The Namespace entry is needed because the applications will create their namespace with CreateNamespace=true. Any other cluster-scoped resource, such as a ClusterRole, is refused. Apply it:
kubectl apply -f project-team-web.yaml
argocd proj get team-web
Step 3 - Deploying to every cluster with an ApplicationSet
The ApplicationSet controller, included in Argo CD, generates Applications from a template. The cluster generator produces one set of parameters per registered cluster that matches a label selector, so a new cluster labeled env=prod automatically receives the application. Create the file:
nano appset-guestbook.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: guestbook
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- clusters:
selector:
matchLabels:
env: prod
template:
metadata:
name: "guestbook-{{.name}}"
spec:
project: team-web
source:
repoURL: https://github.com/argoproj/argocd-example-apps.git
targetRevision: HEAD
path: guestbook
destination:
server: "{{.server}}"
namespace: guestbook
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
{{.name}} and {{.server}} come from each cluster Secret. With missingkey=error, a typo in a parameter fails loudly instead of rendering an empty string. Apply it:
kubectl apply -f appset-guestbook.yaml
Two Applications appear and sync on their own:
argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS
argocd/guestbook-prod-eu https://203.0.113.20:6443 guestbook team-web Synced Healthy Auto-Prune <none>
argocd/guestbook-prod-us https://203.0.113.30:6443 guestbook team-web Synced Healthy Auto-Prune <none>
Verify on one of the target clusters:
kubectl --context prod-eu-context -n guestbook get pods
NAME READY STATUS RESTARTS AGE
guestbook-ui-6b7f6d9874-wz4kq 1/1 Running 0 1m
argocd cluster list now reports both clusters as Successful. Editing the ApplicationSet template updates every generated Application, and deleting the ApplicationSet deletes them, together with their resources.
Step 4 - Restricting the team with RBAC
Argo CD RBAC is configured in the argocd-rbac-cm ConfigMap. Policies grant actions on resources in the form p, subject, resource, action, project/object, effect, and g lines assign users or SSO groups to roles. The following policy gives everyone read-only access by default and lets the web-team group, and a local user alice, sync applications in the team-web project only.
First create the local user by adding it to argocd-cm:
kubectl -n argocd patch configmap argocd-cm --type merge -p '{"data":{"accounts.alice":"login"}}'
Then write the RBAC ConfigMap:
nano argocd-rbac-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-rbac-cm
namespace: argocd
labels:
app.kubernetes.io/name: argocd-rbac-cm
app.kubernetes.io/part-of: argocd
data:
policy.default: role:readonly
scopes: "[groups]"
policy.csv: |
p, role:web-dev, applications, get, team-web/*, allow
p, role:web-dev, applications, sync, team-web/*, allow
p, role:web-dev, logs, get, team-web/*, allow
g, web-team, role:web-dev
g, alice, role:web-dev
This replaces the whole ConfigMap, so merge any policy you already have. Apply it and set a password for alice. The command asks for your admin password first, then the new one:
kubectl apply -f argocd-rbac-cm.yaml
argocd account update-password --account alice
Check the policy without logging in as the user:
argocd admin settings rbac can alice sync applications 'team-web/guestbook-prod-eu' --namespace argocd
argocd admin settings rbac can alice sync applications 'default/some-app' --namespace argocd
Yes
No
Step 5 - Ordering resources with sync waves and hooks
By default Argo CD applies all resources of an application together. Sync waves split them into ordered groups: resources with a lower argocd.argoproj.io/sync-wave value are applied first, and Argo CD waits until they are healthy before moving on. Hooks are resources, usually Jobs, that run at a specific phase such as PreSync or PostSync.
A typical use is running database migrations before the new application version starts. In the Git repository of your application, add a migration Job as a PreSync hook:
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
backoffLimit: 1
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: your_registry/your_app:1.4.0
command: ["./manage.py", "migrate"]
BeforeHookCreation deletes the previous Job before creating a new one on the next sync, so the fixed name never conflicts. If the Job fails, the sync stops and the Deployment is not updated.
Then annotate regular resources with waves. A ConfigMap in wave 0 is applied before the Deployment in wave 1:
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
annotations:
argocd.argoproj.io/sync-wave: "0"
data:
LOG_LEVEL: info
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: your_registry/your_app:1.4.0
envFrom:
- configMapRef:
name: app-config
Commit and push the change, then watch the sync progress through the phases. Replace your_app with the name of the Application:
argocd app sync your_app
argocd app get your_app
The resource list in the output shows the PreSync hook first, then each wave in order.
Step 6 - Sending sync failures to Slack
Argo CD includes a notifications controller configured with the argocd-notifications-cm ConfigMap and the argocd-notifications-secret Secret. Create a Slack app with the chat:write bot scope, install it in your workspace, invite it to the channel (for example #deployments) and copy the bot token. Store the token:
kubectl -n argocd patch secret argocd-notifications-secret --type merge \
-p '{"stringData":{"slack-token":"xoxb-your-slack-bot-token"}}'
Define the Slack service, a message template and a trigger that fires when a sync ends in Error or Failed:
nano argocd-notifications-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
namespace: argocd
data:
service.slack: |
token: $slack-token
template.app-sync-failed: |
message: |
Sync of {{.app.metadata.name}} failed: {{.app.status.operationState.message}}
trigger.on-sync-failed: |
- when: app.status.operationState != nil and app.status.operationState.phase in ['Error', 'Failed']
send: [app-sync-failed]
$slack-token refers to the key you stored in the Secret. Apply the ConfigMap:
kubectl apply -f argocd-notifications-cm.yaml
Applications opt in with an annotation. To subscribe every application generated in Step 3, add it to the ApplicationSet template in appset-guestbook.yaml:
template:
metadata:
name: "guestbook-{{.name}}"
annotations:
notifications.argoproj.io/subscribe.on-sync-failed.slack: deployments
Apply the ApplicationSet again with kubectl apply -f appset-guestbook.yaml. To test, point one application at a path that does not exist, sync it and check the channel. If nothing arrives, read the controller logs:
kubectl -n argocd logs deploy/argocd-notifications-controller --tail=50
Troubleshooting
The ApplicationSet creates no Applications. The selector matches no cluster Secret, or the template failed to render. Check the conditions with kubectl -n argocd get applicationset guestbook -o yaml and the controller logs with kubectl -n argocd logs deploy/argocd-applicationset-controller.
A cluster shows Failed with a connection error. The API server address stored for the cluster is not reachable from the Argo CD pods, often because the kubeconfig used 127.0.0.1 or a private name. Remove it with argocd cluster rm your_server_url and add it again from a kubeconfig that uses an address reachable from the management cluster.
Sync fails with resource ... is not permitted in project. The project does not allow that destination namespace or cluster-scoped kind. Add it to destinations or clusterResourceWhitelist in the AppProject.
Application stays OutOfSync after a push. Argo CD polls Git every three minutes by default. Force a check with argocd app get your_app --hard-refresh, or configure a webhook from your Git provider.
Conclusion
You registered two clusters in Argo CD, deployed the same application to both with a label-driven ApplicationSet, confined a team to one project with RBAC, ordered a release with a pre-sync migration hook and waves, and routed sync failures to Slack. As next steps, you can use a matrix generator to combine clusters with a Git directory generator, manage secrets in Git with Sealed Secrets or External Secrets Operator, or add Argo Rollouts for canary releases.