Setup local dev environment for Argo Workflows with k3s and k3d

Local dev environment for Argo Workflows can help us to develop and test workflow configurations however setting up Argo Workflows can be a pain at times. To address this issue, we will be looking into how we can setup a lightweight Kubernetes cluster using k3s and k3d and running Argo Workflows in our local machine.

What is Argo Workflows?

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).

• Define workflows where each step in the workflow is a container.
• Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a directed acyclic graph (DAG).
• Easily run compute intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes.

Argo is a Cloud Native Computing Foundation (CNCF) hosted project.

Setup k3s cluster

Follow the instruction from my previous post on setting up local dev kubernetes cluster based on docker, k3s, k3d and setup a dev kuberenetes cluster.

https://suren-1988.medium.com/setup-your-personal-kubernetes-cluster-with-k3s-and-k3d-7979976cde5

Setup Argo Workflows

You can choose one of two common installations

Cluster wide installation

Controller operates on all namespaces. ClusterRoles and CulsterRoleBinding are created to access the resources across the namespaces. Workflows can be launched into different namespaces.

# Create a new namespace named argo
kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/install.yaml

For this post, I am using the above option.

You can also install a specific version of the argo.

# Create a new namespace named argo
kubectl create namespace argo

# Install argo v2.9.3 to new namespace
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/v2.9.3/manifests/install.yaml

Namespace installation

Installation of argo which operates on a single namespace. Controller will be configured to run with Role and RoleBinding to access resources from a specific namespace. Install to argo namespace as an example. Workflows can be executed in that specific namespace.

# Create a new namespace named argo
kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/namespace-install.yaml

For more options on the installation, refer to the official documentation.

Workflow pods where getting stuck waiting on MountVolume when I ran the workflows in k3s cluster managed by k3d.

Warning Unable to attach or mount volumes: unmounted volumes=[docker-sock], unattached volumes=[podmetadata docker-sock my-minio-cred default-token-rgwgn]: timed out waiting for the condition
 
Warning FailedMount 114s (x10 over 6m3s) kubelet, k3d-dev-cluster-server-0 MountVolume.SetUp failed for volume “docker-sock” : hostPath type check failed: /var/run/docker.sock is not a socket file

Configuring the workflow executor’s containerRuntimeExecutoras k8sapihas solved the problem.

# Create a config map to configure executor
k8sapi-executor.yml:
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  config: |
    containerRuntimeExecutor: k8sapi
 
kubectl apply -n argo -f k8sapi-executor.yml

A workflow executor is a process that conforms to a specific interface that allows Argo to perform certain actions like monitoring pod logs, collecting artifacts, managing container lifecycles, etc..

The executor to be used in your workflows can be changed in the configmap under the containerRuntimeExecutor key.

To know more about workflow executor, refer the official documentation.

Access Argo Server Dashboard

Let’s access the argo workflows UI by doing the port forward

kubectl -n argo port-forward deployment/argo-server 2746:2746

This will serve the user interface on https://localhost:2746

Deploy first workflow

The Workflow is the most important resource in Argo and serves two important functions:

1. It defines the workflow to be executed.
2. It stores the state of the workflow.

Workflow RBAC

All pods in a workflow run with the service account specified in workflow.spec.serviceAccountName, or if omitted, the default service account of the workflow's namespace. The amount of access which a workflow needs is dependent on what the workflow needs to do. For example, if your workflow needs to deploy a resource, then the workflow's service account will require 'create' privileges on that resource.

Following configuration should be applied for a bare minimum for a workflow to function

workflow_rbac.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
  name: workflow
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-role
rules:
# pod get/watch is used to identify the container IDs of the current pod
# pod patch is used to annotate the step's outputs back to controller (e.g. artifact location)
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - watch
  - patch
# logs get/watch are used to get the pods logs for script outputs, and for log archival
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: workflow-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: workflow-role
subjects:
- kind: ServiceAccount
  name: workflow
kubectl apply -f workflow_rbac.yaml

Lets define our first workflow with serviceAccountName as workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  namespace: default          # Create the Workflow in "default" namespace
  generateName: hello-world-  # Name of this Workflow
spec:
  serviceAccountName: workflow
  entrypoint: whalesay        # Defines "whalesay" as the "main" template
  templates:
  - name: whalesay            # Defining the "whalesay" template
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hello world"]

We can view the execution details from the argo UI.

You can also view the corresponding kubernetes using kubectl commands.

kubectl get po
NAME                READY   STATUS      RESTARTS   AGE
hello-world-dcklg   0/2     Completed   0          7m21s
kubectl logs hello-world-dcklg main
 _____________ 
< hello world >
 ------------- 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/