Data/ML Engineer Blog

Argo Workflows

Argo Workflows: Kubernetes-Native Workflow Orchestration Explained

Introduction

Running workflows on Kubernetes used to be painful. You’d write scripts, build Docker images, create Pod specs, handle failures manually, and pray everything worked together.

Argo Workflows changed this. It’s a workflow engine built specifically for Kubernetes. Each step in your workflow runs as a container. Dependencies are explicit. Retries are automatic. The whole thing is defined in YAML and lives in your cluster.

This isn’t another tool trying to fit into Kubernetes. Argo was designed for Kubernetes from day one. If you’re already running containers in production, Argo gives you a native way to orchestrate them without bolting on external systems.

This guide covers what Argo Workflows actually is, when it makes sense, and how it compares to alternatives. You’ll learn the core concepts, see real patterns, and understand where it fits in modern data and ML infrastructure.

What is Argo Workflows?

Argo Workflows is a container-native workflow engine for Kubernetes. It’s part of the Cloud Native Computing Foundation (CNCF) and graduated to full project status in 2022.

The basic idea is simple. You define workflows as Kubernetes custom resources. Each step runs in its own container. Argo handles scheduling, dependency management, retries, and monitoring. Everything runs inside your cluster.

Unlike traditional workflow engines that run separately and call into Kubernetes, Argo is a Kubernetes application through and through. It uses native Kubernetes resources. It scales with your cluster. It integrates with existing K8s tooling.

The project started at Applatix in 2017 and was open-sourced early. Intuit acquired Applatix and contributed Argo to the CNCF. Today, companies like Google, NVIDIA, Adobe, and Datadog use it in production.

Core Concepts

Understanding Argo means understanding a few key concepts.

Workflows are the top-level resource. A workflow defines the steps to execute and how they relate. You write workflows in YAML and submit them to Kubernetes like any other resource.

Templates define what each step does. The most common is a container template, which runs a Docker image. But Argo also supports script templates (inline scripts), resource templates (create K8s resources), and suspend templates (pause execution).

Steps define the execution order. You can run steps sequentially, in parallel, or using complex DAG patterns. Dependencies determine when each step runs.

Artifacts handle data passing between steps. One step outputs data, another consumes it. Argo can store artifacts in S3, GCS, or any S3-compatible storage.

Parameters let you pass values into workflows and between steps. They make workflows reusable and dynamic.

A Simple Example

Here’s what a basic Argo workflow looks like:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: step1
        template: whalesay
        arguments:
          parameters:
          - name: message
            value: "hello world"
  
  - name: whalesay
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]

This workflow has two templates. The main template defines steps. The whalesay template runs a container. When you submit this, Argo creates a Pod, runs the container, and captures the output.

When Argo Makes Sense

Argo isn’t for everyone. It shines in specific scenarios.

You’re already on Kubernetes. This is the big one. If you’re not running Kubernetes, Argo adds massive overhead. But if you are, Argo is a natural fit.

Your workflows are container-based. Each step runs in a container. If your work fits this model, Argo works well. Data processing, ML training, image rendering, video encoding all work great.

You need to orchestrate Kubernetes resources. Beyond just running containers, Argo can create Jobs, Services, or custom resources as workflow steps. This is powerful for complex deployments or testing scenarios.

You want infrastructure as code. Workflows are defined in YAML and version controlled. They deploy like any Kubernetes resource. This fits GitOps workflows naturally.

You need fine-grained resource control. Every step can specify its own CPU, memory, GPU requirements. You can use node selectors, tolerations, and affinity rules per step.

Common Use Cases

Data Processing Pipelines

Argo handles batch data processing well. Each step processes a chunk of data and passes results to the next step.

A typical pattern: extract data from a source, transform it through multiple stages, load it into a warehouse. Each stage runs in its own container with specific resource requirements.

The artifact system makes this clean. One step writes to S3, the next reads from it. Argo handles the plumbing.

Machine Learning Workflows

ML teams use Argo extensively. Training workflows often have multiple stages: data preparation, feature engineering, model training, evaluation, deployment.

Each stage can use different container images. Data prep might use pandas and sklearn. Training uses TensorFlow or PyTorch with GPU support. Evaluation runs on CPU. Argo handles the transitions.

Hyperparameter tuning fits naturally. Run training with different parameters in parallel. Argo manages the parallelism and collects results.

CI/CD Pipelines

Some teams use Argo for continuous integration and deployment. Build code, run tests, build images, deploy to staging, run integration tests, deploy to production.

Argo Workflows pairs well with Argo CD (the GitOps deployment tool). Workflows handle build and test. Argo CD handles deployment. Together they cover the full pipeline.

Batch Job Orchestration

Any batch processing workload fits. ETL jobs, report generation, data backfills, periodic cleanup tasks.

The Kubernetes-native approach means you can scale jobs easily. Need more workers? Increase parallelism. Jobs automatically get scheduled across your cluster.

Architecture and Components

Argo Workflows has a simple architecture.

The Workflow Controller is the brain. It runs in your cluster and watches for workflow resources. When you submit a workflow, the controller executes it. It creates Pods for each step, monitors them, handles retries, and updates workflow status.

The Argo Server provides the UI and API. It’s optional but recommended. The UI shows running workflows, lets you view logs, and helps with debugging. The API lets you submit workflows programmatically.

The CLI is how you interact with Argo from the command line. Submit workflows, check status, view logs, delete old runs. The CLI talks to the Kubernetes API or Argo Server.

Everything is stored in Kubernetes. Workflow definitions are custom resources. Status and metadata live in etcd (Kubernetes’s backing store). Completed workflow data can be archived to a database for long-term storage.

Workflow Patterns

Argo supports several execution patterns.

Sequential Steps

The simplest pattern. Run step A, then B, then C. Each waits for the previous to complete.

steps:
- - name: step1
    template: process-data
- - name: step2
    template: validate-data
- - name: step3
    template: load-data

Parallel Execution

Run multiple steps at once. Useful when steps are independent.

steps:
- - name: process-us
    template: process-region
    arguments:
      parameters:
      - name: region
        value: "us"
  - name: process-eu
    template: process-region
    arguments:
      parameters:
      - name: region
        value: "eu"

Both steps run simultaneously. Argo creates Pods for each and runs them in parallel.

DAG Workflows

Directed Acyclic Graphs give you full control over dependencies. Specify exactly what depends on what.

dag:
  tasks:
  - name: A
    template: task-template
  - name: B
    dependencies: [A]
    template: task-template
  - name: C
    dependencies: [A]
    template: task-template
  - name: D
    dependencies: [B, C]
    template: task-template

Task A runs first. B and C run in parallel after A completes. D waits for both B and C.

Conditional Execution

Run steps based on conditions. Check exit codes, parameter values, or expression results.

steps:
- - name: check
    template: run-check
- - name: success-case
    template: handle-success
    when: "{{steps.check.outputs.result}} == 'success'"
  - name: failure-case
    template: handle-failure
    when: "{{steps.check.outputs.result}} == 'failure'"

Loops and Recursion

Iterate over lists or recurse dynamically. Generate steps based on data.

steps:
- - name: process-files
    template: process-file
    arguments:
      parameters:
      - name: file
        value: "{{item}}"
    withItems:
    - file1.txt
    - file2.txt
    - file3.txt

This creates three parallel steps, one for each file.

Artifact Management

Artifacts are how data moves between workflow steps. Argo supports several storage backends.

S3 is the most common. Compatible with AWS S3, MinIO, or any S3-compatible storage. Steps output to S3, other steps read from it.

GCS for Google Cloud Storage. Same pattern, different backend.

HTTP for reading artifacts from URLs. Useful for pulling data from external sources.

Git for checking out repositories as artifacts. Common in CI/CD workflows.

A step outputting an artifact looks like this:

outputs:
  artifacts:
  - name: result
    path: /tmp/output.json
    s3:
      key: results/{{workflow.name}}/output.json

A step consuming it:

inputs:
  artifacts:
  - name: data
    path: /tmp/input.json
    s3:
      key: results/{{workflow.name}}/output.json

Argo handles upload and download automatically.

Resource Management

Every workflow step can specify resource requirements.

container:
  image: my-processor:latest
  resources:
    requests:
      memory: "2Gi"
      cpu: "1000m"
    limits:
      memory: "4Gi"
      cpu: "2000m"

This helps Kubernetes schedule Pods efficiently. Steps with high memory needs get placed on appropriate nodes. GPU workloads go to GPU nodes.

You can use node selectors, tolerations, and affinity rules too.

nodeSelector:
  disktype: ssd
tolerations:
- key: "gpu"
  operator: "Exists"
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu-type
          operator: In
          values:
          - nvidia-tesla-v100

This level of control lets you optimize costs. Run cheap steps on spot instances. Run critical steps on stable nodes. Use expensive resources only when needed.

Error Handling and Retries

Workflows fail. Containers crash. Nodes die. Argo handles this gracefully.

Automatic retries are built in. Set a retry policy on steps or the whole workflow.

retryStrategy:
  limit: 3
  retryPolicy: Always
  backoff:
    duration: "1m"
    factor: 2
    maxDuration: "10m"

This retries up to 3 times with exponential backoff.

Exit handlers let you run cleanup steps regardless of success or failure.

onExit: cleanup

The cleanup template runs when the workflow exits, whether it succeeds or fails.

Continue on failure lets workflows proceed even if some steps fail.

continueOn:
  failed: true

Useful for workflows where some failures are acceptable.

Monitoring and Observability

The Argo UI shows workflow status in real time. You see which steps are running, completed, or failed. Click on a step to see logs, resource usage, and timing information.

The workflow list shows all workflows with filtering and sorting. Search by name, status, or labels.

Argo exposes Prometheus metrics. Track workflow submissions, completions, failures, and execution time. Set up alerts for critical workflows.

Logs from each container are accessible through the UI or CLI. You can stream logs in real time or view historical logs.

Events are emitted for workflow state changes. Integrate with monitoring systems to track workflow health.

Scaling and Performance

Argo scales horizontally. The workflow controller can run in high availability mode with multiple replicas. Work is distributed across controller instances.

The controller watches for workflow resources and processes them. At high scale, you might have thousands of concurrent workflows. The controller handles this by processing workflows in batches and optimizing Kubernetes API calls.

Large workflows with hundreds or thousands of steps work, but they consume resources. The controller keeps workflow state in memory while processing. Very large workflows might need controller tuning.

Completed workflows accumulate over time. Argo can archive them to a PostgreSQL or MySQL database. This keeps the Kubernetes API server from getting overloaded with old workflow objects.

Integration with Other Tools

Argo plays well with the ecosystem.

Argo Events triggers workflows based on events. Webhook received? Start a workflow. File appears in S3? Process it. Message arrives on Kafka? Handle it. Events bridges Argo Workflows with event-driven architectures.

Argo CD handles GitOps deployments. Workflows can trigger CD pipelines. CD can call workflows. Together they cover build, test, and deploy.

Kubeflow uses Argo as its workflow engine. Kubeflow Pipelines compiles to Argo workflows under the hood.

Tekton is an alternative CI/CD system. Some teams use Tekton for CI and Argo for other workflows. They coexist fine.

Prometheus and Grafana monitor Argo. Collect metrics, build dashboards, set alerts.

External systems can submit workflows via the API. Trigger workflows from Airflow, Jenkins, GitHub Actions, or custom applications.

Comparison with Alternatives

Argo vs Airflow

Airflow is the incumbent in data engineering. It’s Python-based with a large ecosystem.

Argo is Kubernetes-native. Airflow runs anywhere but integrating with Kubernetes requires the KubernetesPodOperator, which adds abstraction layers.

Airflow has better data lineage and scheduling features. Argo is simpler for container-based workflows.

Use Airflow for traditional ETL with lots of database connections. Use Argo for containerized data processing on Kubernetes.

Argo vs Prefect

Prefect is a modern Python workflow engine. It focuses on developer experience.

Argo requires Kubernetes. Prefect can run anywhere.

Prefect’s Python API is cleaner than YAML. But YAML is version controllable and declarative.

Prefect is better for data science teams that want Python everywhere. Argo is better for platform teams building infrastructure.

Argo vs Tekton

Tekton is another Kubernetes-native workflow engine focused on CI/CD.

Tekton has stronger opinions about how to structure pipelines. Argo is more flexible.

Tekton integrates well with cloud-native buildpacks and container registries. Argo is more general purpose.

For pure CI/CD, Tekton might be better. For general workflow orchestration, Argo has more features.

Argo vs AWS Step Functions

Step Functions is serverless workflow orchestration on AWS.

Step Functions is fully managed. Argo requires operating a Kubernetes cluster.

Step Functions is locked to AWS. Argo runs anywhere Kubernetes does.

Step Functions pricing is per state transition. Argo is free (aside from infrastructure costs).

Use Step Functions for AWS-native workflows where you want zero operations. Use Argo for portable, container-based workflows.

Challenges and Limitations

Argo isn’t perfect. Several pain points come up.

YAML verbosity is the most common complaint. Workflows can get long. Repetitive configuration is hard to avoid. Some teams build abstractions or use tools like Helm to generate Argo workflows.

Kubernetes requirement is a blessing and curse. You need to run Kubernetes. If you’re not already, that’s a huge overhead. Small teams might find this prohibitive.

Debugging complex workflows can be hard. When something fails deep in a DAG, figuring out why takes effort. The UI helps but isn’t perfect.

Artifact management adds complexity. You need to set up S3 or similar storage. Passing large artifacts between steps can be slow.

Learning curve is real. Understanding Kubernetes concepts is required. Template syntax takes time to master.

No built-in scheduling exists. Argo runs workflows when you submit them. For scheduled execution, you need CronWorkflows or external schedulers.

Best Practices

Here’s what works well in production.

Keep workflows focused. Don’t create mega-workflows that do everything. Break them into smaller, composable pieces.

Use templates as building blocks. Create reusable templates for common operations. Reference them from multiple workflows.

Version control everything. Workflows are code. Store them in Git. Use CI/CD to deploy them.

Set resource limits. Always specify requests and limits. This prevents resource starvation and helps with cost optimization.

Implement proper error handling. Use retries, exit handlers, and continue-on-failure strategically. Don’t just let things silently fail.

Archive old workflows. Don’t let completed workflows accumulate forever. Set up archival to a database.

Monitor workflow metrics. Track success rates, execution time, and failure reasons. Set up alerts for critical workflows.

Use artifacts carefully. Large artifacts slow things down. Consider streaming data through external systems instead of passing through artifacts.

Test workflows in isolation. Submit test runs with dry-run or different parameters before running in production.

Document workflow logic. Add descriptions to workflows and templates. Future maintainers will thank you.

Getting Started

Setting up Argo is straightforward if you have Kubernetes.

Install the workflow controller:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/latest/download/install.yaml

Install the CLI:

brew install argo  # macOS
# or download from GitHub releases

Submit your first workflow:

argo submit -n argo --watch hello-world.yaml

Access the UI:

kubectl -n argo port-forward deployment/argo-server 2746:2746

Open your browser to https://localhost:2746.

From here, you can explore examples, build your own workflows, and integrate with your systems.

Real-World Adoption

Many companies run Argo in production.

Google Cloud uses Argo for internal workflows and offers it as part of GKE.

NVIDIA runs ML training workflows at scale with Argo.

Datadog processes millions of data points daily through Argo workflows.

Adobe uses Argo for batch processing and ETL pipelines.

Tesla leverages Argo for data processing related to autonomous driving.

The CNCF ecosystem has embraced Argo. It’s a graduated project, indicating maturity and stability.

The Future Direction

The Argo project continues to evolve. Several areas are getting attention.

Better UI and UX improvements are ongoing. Making workflows easier to visualize and debug.

Performance optimizations for very large workflows. Handling tens of thousands of concurrent workflows efficiently.

Enhanced artifact support with new storage backends and better performance.

Improved observability with better metrics, logs, and tracing integration.

Ecosystem integrations with other CNCF projects and cloud platforms.

The community is active. Monthly releases add features and fixes. Enterprise support is available from multiple vendors.

Key Takeaways

Argo Workflows is a powerful tool for container-based workflow orchestration on Kubernetes.

It’s not for everyone. If you’re not on Kubernetes, the barrier to entry is high. But if you are, Argo is a natural fit.

The container-native approach gives fine-grained control over resources. Each step runs in its own container with specific requirements.

Common use cases include data processing, ML workflows, CI/CD, and batch jobs. Any containerized workload fits the model.

Challenges include YAML verbosity, debugging complexity, and the learning curve. But teams that invest in Argo find it pays off.

Start simple. Run basic workflows first. Build up complexity gradually. Use best practices from the community.

Argo is production-ready. Many large companies trust it for critical workflows. The CNCF graduation status indicates maturity.

If you’re building workflows on Kubernetes, give Argo serious consideration. It might be exactly what you need.

Tags: Argo Workflows, Kubernetes orchestration, container workflows, workflow engine, Kubernetes-native, CNCF, cloud-native workflows, MLOps, data pipelines, workflow automation, CI/CD pipelines, DAG workflows, container orchestration, batch processing, DevOps tools