Argo CD Advanced Configuration and Multi-Cluster

Argo CD is a declarative GitOps continuous delivery tool for Kubernetes, and its advanced features enable scalable multi-cluster deployments, progressive delivery, and robust RBAC for large teams. By leveraging ApplicationSets, sync waves, hooks, and notification systems, you can manage hundreds of applications across multiple clusters from a single control plane.

Prerequisites

  • Argo CD installed on a management cluster (v2.8+)
  • Access to target clusters (kubeconfig)
  • argocd CLI installed
  • Git repository with Kubernetes manifests
  • For progressive delivery: Argo Rollouts installed on target clusters

Multi-Cluster Setup

Register target clusters with the Argo CD control plane:

# Login to Argo CD
argocd login argocd.example.com --username admin

# Add a cluster (uses current kubeconfig context)
argocd cluster add staging-cluster \
  --name staging \
  --namespace argocd

argocd cluster add production-cluster \
  --name production \
  --namespace argocd

# List registered clusters
argocd cluster list

# Check cluster connectivity
argocd cluster get production

For clusters without direct network access, use the Argo CD agent model:

# Install argocd-agent on the remote cluster
kubectl --context=remote-cluster create namespace argocd-agent

# Create a service account and RBAC for the agent
cat > argocd-agent-rbac.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argocd-agent
  namespace: argocd-agent
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd-agent
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argocd-agent
subjects:
  - kind: ServiceAccount
    name: argocd-agent
    namespace: argocd-agent
EOF

kubectl --context=remote-cluster apply -f argocd-agent-rbac.yaml

ApplicationSets for Scalable Deployments

ApplicationSets automate the creation of multiple Argo CD Applications:

# Deploy to all clusters from a list generator
cat > appset-clusters.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-all-clusters
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production
  template:
    metadata:
      name: '{{name}}-guestbook'
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/gitops-repo
        targetRevision: HEAD
        path: 'apps/guestbook/overlays/{{metadata.labels.environment}}'
      destination:
        server: '{{server}}'
        namespace: guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

kubectl apply -f appset-clusters.yaml

# Git directory generator - one app per directory
cat > appset-git-dirs.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/your-org/cluster-apps
        revision: HEAD
        directories:
          - path: "apps/*"
          - path: "apps/excluded-app"
            exclude: true
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/cluster-apps
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
EOF

kubectl apply -f appset-git-dirs.yaml

# Matrix generator - all apps on all clusters
cat > appset-matrix.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: all-apps-all-clusters
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  managed: "true"
          - git:
              repoURL: https://github.com/your-org/apps
              revision: HEAD
              files:
                - path: "apps/*/config.json"
  template:
    metadata:
      name: '{{cluster.name}}-{{appName}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/apps
        targetRevision: HEAD
        path: '{{appPath}}'
      destination:
        server: '{{cluster.server}}'
        namespace: '{{namespace}}'
EOF

kubectl apply -f appset-matrix.yaml

Sync Waves and Hooks

Control deployment order with sync waves and resource hooks:

# Deploy resources in order using sync waves
# Lower wave numbers deploy first
cat > ordered-deploy.yaml <<EOF
# Wave 0: CRDs and namespaces
apiVersion: v1
kind: Namespace
metadata:
  name: myapp
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Wave 1: Secrets and ConfigMaps
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: myapp
  annotations:
    argocd.argoproj.io/sync-wave: "1"
data:
  app.conf: |
    debug=false
---
# Wave 2: Database
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: myapp
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  # ...
---
# Wave 3: Application (waits for DB to be ready)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: myapp
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  # ...
EOF

# Pre-sync hook - runs before sync (e.g., DB migration)
cat > pre-sync-hook.yaml <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  namespace: myapp
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: your-app:latest
          command: ["python", "manage.py", "migrate"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: url
EOF

# Post-sync hook - runs after sync (e.g., smoke tests)
cat > post-sync-hook.yaml <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: test
          image: curlimages/curl
          command: ["curl", "-f", "http://webapp-service/health"]
EOF

RBAC Configuration

Configure granular access control for teams:

# Edit ArgoCD RBAC ConfigMap
kubectl -n argocd edit configmap argocd-rbac-cm

# Or apply a ConfigMap with RBAC rules
cat > argocd-rbac.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # DevOps team - admin access to all projects
    p, role:devops, applications, *, */*, allow
    p, role:devops, clusters, *, *, allow
    p, role:devops, repositories, *, *, allow
    p, role:devops, exec, create, */*, allow

    # Dev team - deploy to staging only
    p, role:developer, applications, get, staging/*, allow
    p, role:developer, applications, sync, staging/*, allow
    p, role:developer, applications, create, staging/*, allow
    p, role:developer, applications, update, staging/*, allow

    # Readonly for all authenticated users
    p, role:readonly, applications, get, */*, allow
    p, role:readonly, clusters, get, *, allow

    # Group assignments
    g, devops-team, role:devops
    g, dev-team, role:developer

  scopes: '[groups]'
EOF

kubectl apply -f argocd-rbac.yaml

# Create a project to scope team access
cat > team-project.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-platform
  namespace: argocd
spec:
  description: Platform team applications
  sourceRepos:
    - https://github.com/your-org/platform-apps
  destinations:
    - namespace: platform-*
      server: https://production-cluster-api:6443
    - namespace: platform-*
      server: https://staging-cluster-api:6443
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
  roles:
    - name: project-admin
      description: Full access to platform project
      policies:
        - p, proj:team-platform:project-admin, applications, *, team-platform/*, allow
      groups:
        - platform-team
EOF

kubectl apply -f team-project.yaml

Notifications and Alerts

Set up Argo CD notifications for Slack, email, and webhooks:

# Install Argo CD Notifications
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/release-1.0/manifests/install.yaml

# Configure Slack integration
kubectl -n argocd patch cm argocd-notifications-cm \
  --type merge \
  -p '{"data": {"service.slack": "{ token: $slack-token }" }}'

kubectl -n argocd create secret generic argocd-notifications-secret \
  --from-literal=slack-token=xoxb-your-slack-token

# Create notification template and trigger
cat > notifications-config.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  template.app-deployed: |
    message: |
      Application {{.app.metadata.name}} has been deployed.
      Revision: {{.app.status.sync.revision}}
      Environment: {{.app.metadata.labels.environment}}
    slack:
      attachments: |
        [{
          "color": "#18be52",
          "title": "{{.app.metadata.name}} deployed",
          "text": "Sync to revision {{.app.status.sync.revision}} succeeded."
        }]
  template.app-sync-failed: |
    message: |
      Application {{.app.metadata.name}} sync FAILED.
    slack:
      attachments: |
        [{
          "color": "#E96D76",
          "title": "Sync failed: {{.app.metadata.name}}"
        }]
  trigger.on-deployed: |
    - description: Notify when app is synced and healthy
      when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
      send: [app-deployed]
  trigger.on-sync-failed: |
    - description: Notify when sync fails
      when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]
  defaultTriggers: |
    - on-deployed
    - on-sync-failed
EOF

kubectl apply -f notifications-config.yaml

# Subscribe an application to notifications
kubectl -n argocd annotate application myapp \
  notifications.argoproj.io/subscribe.on-deployed.slack=deployments

Secret Management

Integrate with external secret stores:

# Using Sealed Secrets
# Install kubeseal CLI and the controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.26.0/controller.yaml

# Seal a secret for GitOps storage
kubectl create secret generic db-secret \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-secret.yaml

# Commit sealed-db-secret.yaml to Git - it's safe to store

# Using Argo CD Vault Plugin (argocd-vault-plugin)
cat > avp-patch.yaml <<EOF
data:
  configManagementPlugins: |
    - name: argocd-vault-plugin
      generate:
        command: ["argocd-vault-plugin"]
        args: ["generate", "./"]
EOF

kubectl -n argocd patch cm argocd-cm --type merge --patch-file avp-patch.yaml

# In your manifests, reference Vault secrets with placeholders:
# password: <path:secret/data/myapp#password>

Progressive Delivery Patterns

Use Argo Rollouts for canary and blue-green deployments:

# Install Argo Rollouts
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f \
  https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# Canary rollout strategy
cat > canary-rollout.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: webapp
  namespace: production
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: webapp-canary
      stableService: webapp-stable
      trafficRouting:
        nginx:
          stableIngress: webapp-ingress
      steps:
        - setWeight: 10
        - pause: {duration: 5m}  # Wait 5 minutes at 10% traffic
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {duration: 10m}
        - setWeight: 100
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
        - name: webapp
          image: your-app:latest
EOF

kubectl apply -f canary-rollout.yaml

# Monitor rollout progress
kubectl argo rollouts get rollout webapp -n production --watch

Troubleshooting

Application stuck in OutOfSync:

# Force refresh to re-evaluate state
argocd app get myapp --refresh

# Compare live vs desired state
argocd app diff myapp

# Check for ignored differences in app spec
kubectl -n argocd get application myapp -o yaml | grep -A10 ignoreDifferences

Sync failing with resource errors:

# View sync operation details
argocd app sync-windows ls myapp

# Check operation logs
argocd app logs myapp

# View events in target namespace
kubectl -n myapp events --sort-by=.lastTimestamp

ApplicationSet not generating applications:

# Check ApplicationSet controller logs
kubectl -n argocd logs deploy/argocd-applicationset-controller

# Validate generator configuration
kubectl -n argocd describe applicationset my-appset

Conclusion

Argo CD's advanced features transform it from a single-app CD tool into a full GitOps platform capable of managing hundreds of applications across dozens of clusters. ApplicationSets eliminate manual application creation, sync waves enforce deployment ordering, and RBAC enables safe multi-team access. Combined with Argo Rollouts for progressive delivery and notifications for observability, you have a complete GitOps platform for production Kubernetes.