Kubernetes multicluster add-on bootstrapping at scale with GitOps using ArgoCD’s ApplicationSet Controller

What is Cluster GitOps and ArgoCD?

ArgoCD Workflow
Muticluster GitOps with ArgoCD

Environment Setup

Target Multi-cluster Local Setup
# My host IP is 10.76.111.19
# Replace the host IP with yours
# Using 6443 as API port for cluster1
k3d cluster create cluster1 --port 8080:80@loadbalancer --port 8443:443@loadbalancer --api-port 10.76.111.19:6443 --k3s-server-arg --tls-san="10.76.111.19"# Using 6444 as API port for cluster1# Replace the host IP 10.76.111.19 with yoursk3d cluster create cluster2 --api-port 10.76.111.19:6444 --k3s-server-arg --tls-san="10.76.111.19"# Confirm the cluster by viewing the configurationkubectl config view#Above command should return something similar to belowapiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://10.76.111.19:6443
name: k3d-cluster1
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://10.76.111.19:6444
name: k3d-cluster2
contexts:
- context:
cluster: k3d-cluster1
user: admin@k3d-cluster1
name: k3d-cluster1
- context:
cluster: k3d-cluster2
user: admin@k3d-cluster2
name: k3d-cluster2
current-context: k3d-cluster2
kind: Config
preferences: {}
users:
- name: admin@k3d-cluster1
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
- name: admin@k3d-cluster2
user:
client-certificate-data: REDACTED
client-key-data: REDACTED

ArgoCD Installation

# Switch the context to use cluster1
kubectl
config use-context k3d-cluster1
# Download ArgoCD manifest for version 2.0.1wget https://raw.githubusercontent.com/argoproj/argo-cd/v2.0.1/manifests/install.yaml# Edit the downloaded install.yaml in your favourite text editor
# Add --insecure and --rootpath parameters to the argocd-server container...
containers:
- command:
- argocd-server
- --staticassets
- /shared/app
# Add insecure and argocd as rootpath
- --insecure
- --rootpath
- /argocd

image: quay.io/argoproj/argocd:v2.0.1
imagePullPolicy: Always
...
# Create a argocd namespacekubectl create namespace argocd# Deploy ArgoCD resources in argocd namespacekubectl create -n argocd -f install.yaml#Create an Ingress to redirect /argocd to the argocd servicecat > ingress.yaml << EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: argocd-ingress
labels:
app: argocd
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /argocd
backend:
serviceName: argocd-server
servicePort: 80
EOF
#Apply ingress configurationkubectl apply -f ingress.yaml -n argocd
Argo CD Login Page
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo
Applications View in Argo CD UI
Cluster View in Argo CD UI
kubectl port-forward svc/argocd-server -n argocd 8081:443
argocd login localhost:8081WARNING: server is not configured with TLS. Proceed (y/n)? y
Username: admin
Password:
'admin:login' logged in successfully
Context 'localhost:8081' updated
argocd cluster add k3d-cluster2INFO[0000] ServiceAccount "argocd-manager" created in namespace "kube-system" 
INFO[0000] ClusterRole "argocd-manager-role" created
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" created
Cluster 'https://10.76.111.19:6444' added
Cluster View in Argo CD UI

ArgoCD’s ApplicationSet Controller

How ApplicationSet controller interacts with Argo CD

Source: ArgoCD ApplicationSet Documentation

App of Apps pattern vs ApplicationSet

Deploy a Argo Workflows cluster add-on via ApplicationSet controller

# Switch the context to use cluster1
kubectl
config use-context k3d-cluster1
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.1.0/manifests/install.yaml
cat > application-set.yaml << EOF
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: argo-workflows
spec:
generators:
- clusters: {}
template:
metadata:
name: '{{name}}-argo-workflows'
spec:
project: "default"
source:
repoURL: https://github.com/surenraju/argo-workflows-manifest.git
targetRevision: HEAD
path: kubernetes
destination:
server: '{{server}}'
namespace: argo
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
EOF
kubectl create -n argocd -f application-set.yaml
Applications View in Argo CD UI
Argo Workflow Application in Argo CD UI

Access Argo Server Dashboard

kubectl -n argo port-forward deployment/argo-server 2746:2746
Argo Workflows Web UI

--

--

--

Site Reliability Engineer linkedin.com/in/surenraju/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Add a Custom Right-Click Menu to an Access Control

The Software Engineering Interview Study Guide

Introducing New Teammates Dan & Sara

Jenkins setup in google cloud platform!

Bug Management | QA Perspective

Class Activity : Naming

Development of features using Agile Methodology

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Suren Raju

Suren Raju

Site Reliability Engineer linkedin.com/in/surenraju/

More from Medium

Monitoring multiple OKE clusters with Prometheus, Thanos and Grafana — Part 1

Monitor ArgoCD applications using Prometheus and AlertManager

Setup EKS Cluster with Pulumi and Helm

Using Cincinnati Operator (AKA Update-Service-Operator) in disconnected environments