Argo Workflows for Kubernetes Job Orchestration

Argo Workflows is a Kubernetes-native workflow engine for orchestrating parallel jobs, data pipelines, and CI/CD using directed acyclic graphs (DAGs). This guide covers installing Argo Workflows, creating workflow templates, managing artifacts, scheduling cron workflows, and building CI/CD pipelines.

Prerequisites

  • Kubernetes cluster 1.24+
  • kubectl configured with cluster access
  • Helm 3.x
  • S3-compatible storage (MinIO, AWS S3) for artifact management
  • Argo CLI (installed in this guide)

Install Argo Workflows

# Create the namespace
kubectl create namespace argo

# Install Argo Workflows (quick start manifest)
ARGO_VERSION=v3.5.4
kubectl apply -n argo \
  -f https://github.com/argoproj/argo-workflows/releases/download/${ARGO_VERSION}/install.yaml

# Wait for the pods to be ready
kubectl wait --for=condition=Ready pods --all -n argo --timeout=120s

# Verify the installation
kubectl get pods -n argo

# Install the Argo CLI
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/${ARGO_VERSION}/argo-linux-amd64.gz
gunzip argo-linux-amd64.gz
chmod +x argo-linux-amd64
sudo mv argo-linux-amd64 /usr/local/bin/argo
argo version

Access the Argo UI

# Port-forward to the Argo server
kubectl -n argo port-forward svc/argo-server 2746:2746

# The UI is available at https://localhost:2746
# Note: uses self-signed TLS by default

# For a local development cluster (no auth):
kubectl patch deployment argo-server -n argo \
  --patch '{"spec":{"template":{"spec":{"containers":[{"name":"argo-server","args":["server","--auth-mode=server"]}]}}}}'

# Configure Argo CLI to connect
argo auth token   # Get the token for UI login
export ARGO_SERVER='localhost:2746'
export ARGO_SECURE=false   # For dev with no TLS

Your First Workflow

# hello-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-
  namespace: argo
spec:
  entrypoint: hello-world
  templates:
    - name: hello-world
      container:
        image: alpine:3.18
        command: [echo]
        args: ["Hello, Argo Workflows!"]
        resources:
          requests:
            memory: 32Mi
            cpu: 100m

# Submit the workflow
argo submit -n argo hello-workflow.yaml --watch

# List workflows
argo list -n argo

# Get workflow logs
argo logs -n argo @latest

# Delete a workflow
argo delete -n argo hello-XXXXX

Parameterized workflow:

# parameterized-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: param-workflow-
spec:
  entrypoint: process-data
  arguments:
    parameters:
      - name: input-file
        value: "data.csv"
      - name: output-bucket
        value: "my-results"
  templates:
    - name: process-data
      inputs:
        parameters:
          - name: input-file
          - name: output-bucket
      container:
        image: my-org/data-processor:latest
        command: [python, /app/process.py]
        args:
          - --input={{inputs.parameters.input-file}}
          - --output={{inputs.parameters.output-bucket}}

# Override parameters at submit time
argo submit -n argo parameterized-workflow.yaml \
  -p input-file=large-dataset.csv \
  -p output-bucket=results-2024

DAG Workflows

DAGs define dependencies between tasks explicitly:

# dag-pipeline.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-pipeline-
spec:
  entrypoint: data-pipeline
  templates:
    - name: data-pipeline
      dag:
        tasks:
          # Task A: no dependencies (runs first)
          - name: extract
            template: run-extract

          # Task B: depends on A
          - name: transform
            template: run-transform
            dependencies: [extract]
            arguments:
              parameters:
                - name: input
                  value: "{{tasks.extract.outputs.parameters.output-path}}"

          # Task C: depends on A (runs in parallel with B)
          - name: validate
            template: run-validate
            dependencies: [extract]

          # Task D: depends on both B and C
          - name: load
            template: run-load
            dependencies: [transform, validate]

    - name: run-extract
      outputs:
        parameters:
          - name: output-path
            valueFrom:
              path: /tmp/output-path.txt
      container:
        image: my-org/etl:latest
        command: [python, extract.py]

    - name: run-transform
      inputs:
        parameters:
          - name: input
      container:
        image: my-org/etl:latest
        command: [python, transform.py]
        args: ["--input={{inputs.parameters.input}}"]

    - name: run-validate
      container:
        image: my-org/etl:latest
        command: [python, validate.py]

    - name: run-load
      container:
        image: my-org/etl:latest
        command: [python, load.py]

Fan-out / fan-in with items:

templates:
  - name: process-all-files
    dag:
      tasks:
        - name: process-file
          template: process-single-file
          arguments:
            parameters:
              - name: filename
                value: "{{item}}"
          withItems:
            - report-jan.csv
            - report-feb.csv
            - report-mar.csv

  - name: process-single-file
    inputs:
      parameters:
        - name: filename
    container:
      image: my-org/processor:latest
      command: [python, process.py, "{{inputs.parameters.filename}}"]

Workflow Templates and Reuse

WorkflowTemplate resources are reusable workflow definitions stored in the cluster:

# workflow-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: docker-build-push
  namespace: argo
spec:
  templates:
    - name: build-and-push
      inputs:
        parameters:
          - name: image
          - name: tag
          - name: context
            value: "."
      container:
        image: gcr.io/kaniko-project/executor:latest
        command:
          - /kaniko/executor
          - --dockerfile=Dockerfile
          - --context={{inputs.parameters.context}}
          - --destination={{inputs.parameters.image}}:{{inputs.parameters.tag}}
        volumeMounts:
          - name: docker-config
            mountPath: /kaniko/.docker/
      volumes:
        - name: docker-config
          secret:
            secretName: docker-registry-credentials

# Reference the template from another workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: ci-build-
spec:
  entrypoint: build
  templates:
    - name: build
      steps:
        - - name: build-image
            templateRef:
              name: docker-build-push
              template: build-and-push
            arguments:
              parameters:
                - name: image
                  value: my-org/my-app
                - name: tag
                  value: "{{workflow.parameters.git-sha}}"

Artifact Management

Configure artifact storage to pass files between steps:

# Configure MinIO as artifact store
# Apply to argo namespace
kubectl create secret generic minio-secret \
  -n argo \
  --from-literal=accesskey=minioadmin \
  --from-literal=secretkey=minioadmin

kubectl patch configmap workflow-controller-configmap -n argo \
  --patch '{"data":{"artifactRepository":"{\"s3\":{\"bucket\":\"argo-artifacts\",\"endpoint\":\"minio.minio.svc:9000\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"minio-secret\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"minio-secret\",\"key\":\"secretkey\"}}}"}}' 

# Workflow using artifacts
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-pipeline-
spec:
  entrypoint: pipeline
  templates:
    - name: pipeline
      steps:
        - - name: generate
            template: generate-report
        - - name: process
            template: process-report
            arguments:
              artifacts:
                - name: report
                  from: "{{steps.generate.outputs.artifacts.report}}"

    - name: generate-report
      outputs:
        artifacts:
          - name: report
            path: /tmp/report.json
      container:
        image: alpine:3.18
        command: [sh, -c]
        args: ['echo "{\"status\":\"ok\",\"count\":42}" > /tmp/report.json']

    - name: process-report
      inputs:
        artifacts:
          - name: report
            path: /tmp/input-report.json
      container:
        image: python:3.11-alpine
        command: [python3, -c]
        args: ['import json; d=json.load(open("/tmp/input-report.json")); print(d)']

Cron Workflows

# cron-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: nightly-report
  namespace: argo
spec:
  schedule: "0 2 * * *"   # Daily at 2am UTC
  timezone: "UTC"
  concurrencyPolicy: Forbid   # Don't run if previous is still running
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 3
  workflowSpec:
    entrypoint: generate-report
    templates:
      - name: generate-report
        container:
          image: my-org/reporter:latest
          command: [python, generate.py]
          env:
            - name: REPORT_DATE
              value: "{{workflow.scheduledTime}}"

# Manage cron workflows
argo cron list -n argo
argo cron suspend nightly-report -n argo
argo cron resume nightly-report -n argo

# Manually trigger a cron workflow
argo cron get nightly-report -n argo
argo submit -n argo --from=cronwf/nightly-report

CI/CD Pipeline Patterns

# Complete CI pipeline as a WorkflowTemplate
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: ci-pipeline
  namespace: argo
spec:
  entrypoint: ci
  arguments:
    parameters:
      - name: git-repo
      - name: git-branch
        value: main
      - name: image
  templates:
    - name: ci
      dag:
        tasks:
          - name: lint
            template: run-lint
          - name: test
            template: run-tests
            dependencies: [lint]
          - name: build
            template: build-image
            dependencies: [test]
          - name: scan
            template: security-scan
            dependencies: [build]
          - name: deploy-staging
            template: deploy
            dependencies: [scan]
            arguments:
              parameters:
                - name: environment
                  value: staging

Troubleshooting

Workflow stuck in Pending state:

# Check pod events
kubectl describe pod -n argo -l workflows.argoproj.io/workflow=hello-XXXXX

# Common cause: resource requests too high for available nodes
kubectl top nodes

Workflow fails with "permission denied":

# Argo needs a service account with proper RBAC
kubectl create rolebinding argo-default-binding \
  --clusterrole=argo-aggregate-to-admin \
  --serviceaccount=argo:default \
  -n argo

Artifacts not being saved:

# Verify artifact repository config
kubectl get configmap workflow-controller-configmap -n argo -o yaml

# Check MinIO/S3 connectivity from a pod
kubectl run -n argo debug --rm -it --image=alpine -- \
  wget http://minio.minio.svc:9000/minio/health/live

DAG task not running:

# View detailed workflow status
argo get -n argo WORKFLOW_NAME -o yaml | grep -A 10 "nodes:"

# A task won't run if its dependencies failed
# Check dependency task logs:
argo logs -n argo WORKFLOW_NAME -c extract

Conclusion

Argo Workflows provides a powerful, Kubernetes-native platform for orchestrating complex job pipelines, CI/CD workflows, and data processing tasks. Its DAG execution model, artifact management, and template reuse make it suitable for everything from simple cron jobs to multi-stage ML training pipelines. Combined with the Argo Events and Argo CD ecosystem, it forms a complete GitOps-native automation platform.