679 lines
17 KiB
Markdown
679 lines
17 KiB
Markdown
---
|
|
title: "A beautiful GitOps day IV - CD with Flux"
|
|
date: 2023-08-22
|
|
description: "Follow this opinionated guide as starter-kit for your own Kubernetes platform..."
|
|
tags: ["kubernetes", "cd", "flux", "nocode", "n8n", "nocodb"]
|
|
---
|
|
|
|
{{< lead >}}
|
|
Use GitOps workflow for building a production grade on-premise Kubernetes cluster on cheap VPS provider, with complete CI/CD 🎉
|
|
{{< /lead >}}
|
|
|
|
This is the **Part IV** of more global topic tutorial. [Back to guide summary]({{< ref "/posts/10-a-beautiful-gitops-day" >}}) for intro.
|
|
|
|
## Flux
|
|
|
|
In GitOps world, 2 tools are leading for CD in k8s: **Flux** and **ArgoCD**. As Flux is CLI first and more lightweight, it's my personal goto. You may wonder why don't continue with actual k3s Terraform project ?
|
|
|
|
You already noted that by adding more and more Helm dependencies to terraform, the plan time is increasing, as well as the state file. So not very scalable.
|
|
|
|
It's the perfect moment to draw a clear line between **IaC** (Infrastructure as Code) and **CD** (Continuous Delivery). IaC is for infrastructure, CD is for application. So to resume our GitOps stack:
|
|
|
|
1. IaC for Hcloud cluster initialization (*the basement*): **Terraform**
|
|
2. IaC for Kubernetes configuration (*the walls*): **Helm** through **Terraform**
|
|
3. CD for any application deployments (*the furniture*): **Flux**
|
|
|
|
{{< alert >}}
|
|
You can probably eliminate with some efforts the 2nd stack by using both `Kube-Hetzner`, which take care of ingress and storage, and using Flux directly for the remaining helms like database cluster. Or maybe you can also add custom helms to `Kube-Hetzner`.
|
|
But as it's increase complexity and dependencies problem, I prefer personally to keep a clear separation between the middle part and the rest, as it's more straightforward for me. Just a matter of taste 🥮
|
|
{{< /alert >}}
|
|
|
|
### Flux bootstrap
|
|
|
|
Create a dedicated Git repository for Flux somewhere, I'm using GitHub, which with [his CLI](https://cli.github.com/) is just a matter of:
|
|
|
|
```sh
|
|
gh repo create demo-kube-flux --private --add-readme
|
|
gh repo clone demo-kube-flux
|
|
```
|
|
|
|
{{< alert >}}
|
|
Put `--add-readme` option to have a non-empty repo, otherwise Flux bootstrap will give you an error.
|
|
{{< /alert >}}
|
|
|
|
Let's back to `demo-kube-k3s` terraform project and add Flux bootstrap connected to above repository:
|
|
|
|
{{< highlight host="demo-kube-k3s" file="main.tf" >}}
|
|
|
|
```tf
|
|
terraform {
|
|
//...
|
|
|
|
required_providers {
|
|
flux = {
|
|
source = "fluxcd/flux"
|
|
}
|
|
github = {
|
|
source = "integrations/github"
|
|
}
|
|
}
|
|
}
|
|
|
|
//...
|
|
|
|
variable "github_token" {
|
|
sensitive = true
|
|
type = string
|
|
}
|
|
|
|
variable "github_org" {
|
|
type = string
|
|
}
|
|
|
|
variable "github_repository" {
|
|
type = string
|
|
}
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
{{< highlight host="demo-kube-k3s" file="flux.tf" >}}
|
|
|
|
```tf
|
|
github_org = "mykuberocks"
|
|
github_repository = "demo-kube-flux"
|
|
github_token = "xxx"
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
{{< alert >}}
|
|
Create a [Github token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) with repo permissions and add it to `github_token` variable.
|
|
{{< /alert >}}
|
|
|
|
{{< highlight host="demo-kube-k3s" file="flux.tf" >}}
|
|
|
|
```tf
|
|
provider "github" {
|
|
owner = var.github_org
|
|
token = var.github_token
|
|
}
|
|
|
|
resource "tls_private_key" "flux" {
|
|
algorithm = "ECDSA"
|
|
ecdsa_curve = "P256"
|
|
}
|
|
|
|
resource "github_repository_deploy_key" "this" {
|
|
title = "Flux"
|
|
repository = var.github_repository
|
|
key = tls_private_key.flux.public_key_openssh
|
|
read_only = false
|
|
}
|
|
|
|
provider "flux" {
|
|
kubernetes = {
|
|
config_path = "~/.kube/config"
|
|
}
|
|
git = {
|
|
url = "ssh://git@github.com/${var.github_org}/${var.github_repository}.git"
|
|
ssh = {
|
|
username = "git"
|
|
private_key = tls_private_key.flux.private_key_pem
|
|
}
|
|
}
|
|
}
|
|
|
|
resource "flux_bootstrap_git" "this" {
|
|
path = "clusters/demo"
|
|
|
|
components_extra = [
|
|
"image-reflector-controller",
|
|
"image-automation-controller"
|
|
]
|
|
|
|
depends_on = [github_repository_deploy_key.this]
|
|
}
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
Note as we'll use `components_extra` to add `image-reflector-controller` and `image-automation-controller` to Flux, as it will serve us later for new image tag detection.
|
|
|
|
After applying this, use `kg deploy -n flux-system` to check that Flux is correctly installed and running.
|
|
|
|
### Managing secrets
|
|
|
|
As always with GitOps, a secured secrets management is critical. Nobody wants to expose sensitive data in a git repository. An easy to go solution is to use [Bitnami Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets), which will deploy a dedicated controller in your cluster that will automatically decrypt sealed secrets.
|
|
|
|
Open `demo-kube-flux` project and create helm deployment for sealed secret.
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/flux-add-ons/sealed-secrets.yaml" >}}
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: source.toolkit.fluxcd.io/v1beta2
|
|
kind: HelmRepository
|
|
metadata:
|
|
name: sealed-secrets
|
|
namespace: flux-system
|
|
spec:
|
|
interval: 1h0m0s
|
|
url: https://bitnami-labs.github.io/sealed-secrets
|
|
---
|
|
apiVersion: helm.toolkit.fluxcd.io/v2beta1
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: sealed-secrets
|
|
namespace: flux-system
|
|
spec:
|
|
chart:
|
|
spec:
|
|
chart: sealed-secrets
|
|
reconcileStrategy: ChartVersion
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: sealed-secrets
|
|
version: ">=2.12.0"
|
|
interval: 1m
|
|
releaseName: sealed-secrets-controller
|
|
targetNamespace: flux-system
|
|
install:
|
|
crds: Create
|
|
upgrade:
|
|
crds: CreateReplace
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
{{< alert >}}
|
|
Don't touch manifests under `flux-system` folder, as it's managed by Flux itself and overload on each flux bootstrap.
|
|
{{< /alert >}}
|
|
|
|
Then push it and check that sealed secret controller is correctly deployed with `kg deploy sealed-secrets-controller -n flux-system`.
|
|
|
|
Private key is automatically generated, so last step is to fetch the public key. Type this in project root to include it in your git repository:
|
|
|
|
```sh
|
|
kpf svc/sealed-secrets-controller -n flux-system 8080
|
|
curl http://localhost:8080/v1/cert.pem > pub-sealed-secrets.pem
|
|
```
|
|
|
|
{{< alert >}}
|
|
By the way install the client with `brew install kubeseal` (Mac / Linux) or `scoop install kubeseal` (Windows).
|
|
{{< /alert >}}
|
|
|
|
## Install some tools
|
|
|
|
It's now finally time to install some tools to help us in our CD journey.
|
|
|
|
### pgAdmin
|
|
|
|
A 1st good example is typically pgAdmin, which is a web UI for Postgres. We'll use it to manage our database cluster. It requires a local PVC to store its data user and settings.
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/postgres/deploy-pgadmin.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: pgadmin
|
|
namespace: postgres
|
|
spec:
|
|
strategy:
|
|
type: Recreate
|
|
selector:
|
|
matchLabels:
|
|
app: pgadmin
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: pgadmin
|
|
spec:
|
|
securityContext:
|
|
runAsUser: 5050
|
|
runAsGroup: 5050
|
|
fsGroup: 5050
|
|
fsGroupChangePolicy: "OnRootMismatch"
|
|
containers:
|
|
- name: pgadmin
|
|
image: dpage/pgadmin4:latest
|
|
ports:
|
|
- containerPort: 80
|
|
env:
|
|
- name: PGADMIN_DEFAULT_EMAIL
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pgadmin-auth
|
|
key: default-email
|
|
- name: PGADMIN_DEFAULT_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pgadmin-auth
|
|
key: default-password
|
|
volumeMounts:
|
|
- name: pgadmin-data
|
|
mountPath: /var/lib/pgadmin
|
|
volumes:
|
|
- name: pgadmin-data
|
|
persistentVolumeClaim:
|
|
claimName: pgadmin-data
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: pgadmin-data
|
|
namespace: postgres
|
|
spec:
|
|
resources:
|
|
requests:
|
|
storage: 128Mi
|
|
volumeMode: Filesystem
|
|
storageClassName: longhorn
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: pgadmin
|
|
namespace: postgres
|
|
spec:
|
|
selector:
|
|
app: pgadmin
|
|
ports:
|
|
- port: 80
|
|
---
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: pgadmin
|
|
namespace: postgres
|
|
spec:
|
|
entryPoints:
|
|
- websecure
|
|
routes:
|
|
- match: Host(`pgadmin.kube.rocks`)
|
|
kind: Rule
|
|
middlewares:
|
|
- name: middleware-ip
|
|
namespace: traefik
|
|
services:
|
|
- name: pgadmin
|
|
port: 80
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
Here are the secrets to adapt to your needs:
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/postgres/secret-pgadmin.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: pgadmin-auth
|
|
namespace: postgres
|
|
type: Opaque
|
|
data:
|
|
default-email: YWRtaW5Aa3ViZS5yb2Nrcw==
|
|
default-password: YWRtaW4=
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
Now be sure to encrypt it with `kubeseal` and remove original file:
|
|
|
|
```sh
|
|
cat clusters/demo/postgres/secret-pgadmin.yaml | kubeseal --format=yaml --cert=pub-sealed-secrets.pem > clusters/demo/postgres/sealed-secret-pgadmin.yaml
|
|
rm clusters/demo/postgres/secret-pgadmin.yaml
|
|
```
|
|
|
|
{{< alert >}}
|
|
Don't forget to remove the original secret file before commit for obvious reason ! If too late, consider password leaked and regenerate a new one.
|
|
You may use [VSCode extension](https://github.com/codecontemplator/vscode-kubeseal)
|
|
{{< /alert >}}
|
|
|
|
Push it and wait a minute, and go to `pgadmin.kube.rocks` and login with chosen credentials. Now try to register a new server with `postgresql-primary.postgres` as hostname, and the rest with your PostgreSQL credential on previous installation. It should work !
|
|
|
|
{{< alert >}}
|
|
If you won't wait each time after code push, do `flux reconcile kustomization flux-system --with-source` (require `flux-cli`). It also allows easy debugging by printing any syntax error in your manifests.
|
|
{{< /alert >}}
|
|
|
|
You can test the read replica too by register a new server using the hostname `postgresql-read.postgres`. Try to do some update on primary and check that it's replicated on read replica. Any modification on replicas should be rejected as it's on transaction read only mode.
|
|
|
|
It's time to use some useful apps.
|
|
|
|
### n8n
|
|
|
|
Let's try some app that require a bit more configuration and real database connection with n8n, a workflow automation tool.
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/n8n/deploy-n8n.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: n8n
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: n8n
|
|
namespace: n8n
|
|
spec:
|
|
strategy:
|
|
type: Recreate
|
|
selector:
|
|
matchLabels:
|
|
app: n8n
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: n8n
|
|
spec:
|
|
containers:
|
|
- name: n8n
|
|
image: n8nio/n8n:latest
|
|
ports:
|
|
- containerPort: 5678
|
|
env:
|
|
- name: N8N_PROTOCOL
|
|
value: https
|
|
- name: N8N_HOST
|
|
value: n8n.kube.rocks
|
|
- name: N8N_PORT
|
|
value: "5678"
|
|
- name: NODE_ENV
|
|
value: production
|
|
- name: WEBHOOK_URL
|
|
value: https://n8n.kube.rocks/
|
|
- name: DB_TYPE
|
|
value: postgresdb
|
|
- name: DB_POSTGRESDB_DATABASE
|
|
value: n8n
|
|
- name: DB_POSTGRESDB_HOST
|
|
value: postgresql-primary.postgres
|
|
- name: DB_POSTGRESDB_USER
|
|
value: n8n
|
|
- name: DB_POSTGRESDB_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: n8n-db
|
|
key: password
|
|
- name: N8N_EMAIL_MODE
|
|
value: smtp
|
|
- name: N8N_SMTP_HOST
|
|
value: smtp.mailgun.org
|
|
- name: N8N_SMTP_PORT
|
|
value: "587"
|
|
- name: N8N_SMTP_USER
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: n8n-smtp
|
|
key: user
|
|
- name: N8N_SMTP_PASS
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: n8n-smtp
|
|
key: password
|
|
- name: N8N_SMTP_SENDER
|
|
value: n8n@kube.rocks
|
|
volumeMounts:
|
|
- name: n8n-data
|
|
mountPath: /home/node/.n8n
|
|
volumes:
|
|
- name: n8n-data
|
|
persistentVolumeClaim:
|
|
claimName: n8n-data
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: n8n-data
|
|
namespace: n8n
|
|
spec:
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
volumeMode: Filesystem
|
|
storageClassName: longhorn
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: n8n
|
|
namespace: n8n
|
|
labels:
|
|
app: n8n
|
|
spec:
|
|
selector:
|
|
app: n8n
|
|
ports:
|
|
- port: 5678
|
|
---
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: n8n
|
|
namespace: n8n
|
|
spec:
|
|
entryPoints:
|
|
- websecure
|
|
routes:
|
|
- match: Host(`n8n.kube.rocks`)
|
|
kind: Rule
|
|
services:
|
|
- name: n8n
|
|
port: 5678
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
Here are the secrets to adapt to your needs:
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/n8n/secret-n8n-db.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: n8n-db
|
|
namespace: n8n
|
|
type: Opaque
|
|
data:
|
|
password: YWRtaW4=
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/n8n/secret-n8n-smtp.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: n8n-smtp
|
|
namespace: n8n
|
|
type: Opaque
|
|
data:
|
|
user: YWRtaW4=
|
|
password: YWRtaW4=
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
While writing these secrets, create `n8n` DB and set `n8n` user with proper credentials as owner.
|
|
|
|
Then don't forget to seal secrets and remove original files the same way as pgAdmin. Once pushed, n8n should be deploying, automatically migrate the db, and soon after `n8n.kube.rocks` should be available, allowing you to create your 1st account.
|
|
|
|
### NocoDB
|
|
|
|
Let's try a final candidate with NocoDB, an Airtable-like generator for Postgres. It's very similar to n8n.
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/nocodb/deploy-nocodb.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: nocodb
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: nocodb
|
|
namespace: nocodb
|
|
spec:
|
|
strategy:
|
|
type: Recreate
|
|
selector:
|
|
matchLabels:
|
|
app: nocodb
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: nocodb
|
|
spec:
|
|
containers:
|
|
- name: nocodb
|
|
image: nocodb/nocodb:latest
|
|
ports:
|
|
- containerPort: 8080
|
|
env:
|
|
- name: DB_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nocodb-db
|
|
key: password
|
|
- name: DATABASE_URL
|
|
value: postgresql://nocodb:$(DB_PASSWORD)@postgresql-primary.postgres/nocodb
|
|
- name: NC_AUTH_JWT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nocodb-auth
|
|
key: jwt-secret
|
|
- name: NC_SMTP_HOST
|
|
value: smtp.mailgun.org
|
|
- name: NC_SMTP_PORT
|
|
value: "587"
|
|
- name: NC_SMTP_USERNAME
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nocodb-smtp
|
|
key: user
|
|
- name: NC_SMTP_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nocodb-smtp
|
|
key: password
|
|
- name: NC_SMTP_FROM
|
|
value: nocodb@kube.rocks
|
|
volumeMounts:
|
|
- name: nocodb-data
|
|
mountPath: /usr/app/data
|
|
volumes:
|
|
- name: nocodb-data
|
|
persistentVolumeClaim:
|
|
claimName: nocodb-data
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: nocodb-data
|
|
namespace: nocodb
|
|
spec:
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
volumeMode: Filesystem
|
|
storageClassName: longhorn
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: nocodb
|
|
namespace: nocodb
|
|
spec:
|
|
selector:
|
|
app: nocodb
|
|
ports:
|
|
- port: 8080
|
|
---
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: nocodb
|
|
namespace: nocodb
|
|
spec:
|
|
entryPoints:
|
|
- websecure
|
|
routes:
|
|
- match: Host(`nocodb.kube.rocks`)
|
|
kind: Rule
|
|
services:
|
|
- name: nocodb
|
|
port: 8080
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
Here are the secrets to adapt to your needs:
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/nocodb/secret-nocodb-db.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: nocodb-db
|
|
namespace: nocodb
|
|
type: Opaque
|
|
data:
|
|
password: YWRtaW4=
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/nocodb/secret-nocodb-auth.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: nocodb-auth
|
|
namespace: nocodb
|
|
type: Opaque
|
|
data:
|
|
jwt-secret: MDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAw
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
{{< highlight host="demo-kube-flux" file="clusters/demo/nocodb/secret-nocodb-smtp.yaml" >}}
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: nocodb-smtp
|
|
namespace: nocodb
|
|
type: Opaque
|
|
data:
|
|
user: YWRtaW4=
|
|
password: YWRtaW4=
|
|
```
|
|
|
|
{{< /highlight >}}
|
|
|
|
The final process is identical to n8n.
|
|
|
|
## 4th check ✅
|
|
|
|
We now have a functional continuous delivery with some nice no-code tools to play with ! The final missing stack for a production grade cluster is to install a complete monitoring stack, this is the [next part]({{< ref "/posts/15-a-beautiful-gitops-day-5" >}}).
|