okami101/blog
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 46 Wiki Activity

init k8s guide

Browse Source
This commit is contained in:
Adrien Beaudouin
2023-08-21 13:34:06 +02:00
parent 7055fd3c80
commit d22c31787e
3 changed files with 152 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
content/posts/11-build-your-own-kubernetes-cluster-part-2/index.md
View File

@ -74,19 +74,19 @@ We have mainly 2 options :
Here are the pros and cons of each module :
| | [Kube Hetzner](https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud/latest) | [Okami101 K3s](https://registry.terraform.io/modules/okami101/k3s) |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Support** | Strong community | Just intended as a reusable starter-kit |
| **Included helms** | Traefik, Longhorn, Cert Manager, Kured | None, just the K3s initial setup, which can be preferable when managing helms dependencies on separated terraform project |
| **Hetzner integration** | Complete, use [Hcloud Controller](https://github.com/hetznercloud/hcloud-cloud-controller-manager) internally, allowing dynamic Load Balancing, autoscaling, cleaner node deletion | Basic, public Load Balancer is statically managed by the nodepool configuration, no autoscaling support |
| **OS** | openSUSE MicroOS, optimized for container worloads | Debian 11 or Ubuntu 22.04 |
| **Initial setup** | Require packer for initial Snapshot creation, and slower on node creation because Hetnzer don't support it natively | Just about ~2 minutes for all cluster setup |
| **Client support** | POSIX-based OS only, require WSL on Windows | All including Powershell |
| **Internal complexity** | Huge, you can't really put your head inside | Very accessible, easy to extend and fork, better for learning |
| **Upgrade** | You may need to follow new versions regularly | As a simple starter-kit, no need to support all community problems, so very few updates |
| **Quality** | Use many hacks to satisfy all community needs, plenty of remote-exec and file provisioner which is not recommended by HashiCorp themselves | Use standard **cloud-config** for initial provisioning, then **Salt** for cluster OS management |
| **Security** | Needs an SSH private key because of local provisioners, and SSH port opened to every node | Require only public SSH key, minimized opened SSH ports to only controllers, use SSH jump from a controller to access any internal worker node |
| **Reusability** | Vendor locked to Hetzner Cloud | Easy to adapt for a different cloud provider as long as it supports **cloud-config** (as 99% of them) |
| | [Kube Hetzner](https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud/latest) | [Okami101 K3s](https://registry.terraform.io/modules/okami101/k3s) |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Support** | Strong community | Just intended as a reusable starter-kit |
| **Included helms** | Traefik, Longhorn, Cert Manager, Kured | None, just the K3s initial setup, as it's generally preferable to manage this helms dependencies on separated terraform project, allowing easier upgrading |
| **Hetzner integration** | Complete, use [Hcloud Controller](https://github.com/hetznercloud/hcloud-cloud-controller-manager) internally, allowing dynamic Load Balancing, autoscaling, cleaner node deletion | Basic, public Load Balancer is statically managed by the nodepool configuration, no autoscaling support |
| **OS** | openSUSE MicroOS, optimized for container worloads | Debian 11 or Ubuntu 22.04 |
| **Initial setup** | Require packer for initial Snapshot creation, and slower on node creation because Hetnzer don't support it natively | Just about ~2 minutes for all cluster setup |
| **Client support** | POSIX-based OS only, require WSL on Windows | All including Powershell |
| **Internal complexity** | Huge, you can't really put your head inside | Very accessible, easy to extend and fork, better for learning |
| **Upgrade** | You may need to follow new versions regularly | As a simple starter-kit, no need to support all community problems, so very few updates |
| **Quality** | Use many hacks to satisfy all community needs, plenty of remote-exec and file provisioner which is not recommended by HashiCorp themselves | Use standard **cloud-config** for initial provisioning, then **Salt** for cluster OS management |
| **Security** | Needs an SSH private key because of local provisioners, and SSH port opened to every node | Require only public SSH key, minimized opened SSH ports to only controllers, use SSH jump from a controller to access any internal worker node |
| **Reusability** | Vendor locked to Hetzner Cloud | Easy to adapt for a different cloud provider as long as it supports **cloud-config** (as 99% of them) |
So for resume, choose Kube Hetzner module if :

11
content/posts/12-build-your-own-kubernetes-cluster-part-3/index.md
View File

@ -254,7 +254,7 @@ resource "helm_release" "traefik" {
`tlsStore.default.defaultCertificate.secretName` will be used to store the default certificate that will be used for all ingress that don't have a specific certificate.
By default, it will deploy 1 single replica of Traefik. But don't worry, when upgrading, the default update strategy is `RollingUpdate`, so it will be upgraded one by one without downtime.
By default, it will deploy 1 single replica of Traefik. But don't worry, when upgrading, the default update strategy is `RollingUpdate`, so it will be upgraded one by one without downtime. Increment `deployment.replicas` if you need more performance.
### Load balancer
@ -394,7 +394,7 @@ dns_api_token = "xxx"
{{</ highlight >}}
Finally, apply the following Terraform code in order to issue the new wildcard certificate for your domain.
Then we need to create a default `Certificate` k8s resource associated to a valid `ClusterIssuer` resource that will manage its generation. Apply the following Terraform code for issuing the new wildcard certificate for your domain.
{{< highlight file="certificates.tf" >}}
@ -470,6 +470,13 @@ resource "kubernetes_manifest" "tls_certificate" {
{{</ highlight >}}
{{< alert >}}
You can set `acme.privateKeySecretRef.name` to **letsencrypt-staging** for testing purpose and note waste limited LE quota.
Set `privateKey.rotationPolicy` to **Always** to ensure that the certificate will be [renewed automatically](https://cert-manager.io/docs/usage/certificate/) 30 days before expires without downtime.
{{</ alert >}}
In the meantime, go to your DNS provider and add a new `*.kube.rocks` entry pointing to the load balancer IP.
Try `test.kube.rocks` to check certificate validity. If not valid, check the certificate status with `kg cert -n traefik` and get challenge status `kg challenges -n traefik`. The certificate must be in `Ready` state after many minutes.

133
content/posts/13-build-your-own-kubernetes-cluster-part-4/index.md
View File

@ -12,7 +12,134 @@ Be free from AWS/Azure/GCP by building a production grade On-Premise Kubernetes
This is the **Part IV** of more global topic tutorial. [Back to first part]({{< ref "/posts/10-build-your-own-kubernetes-cluster" >}}) for intro.
## Resilient Storage with Longhorn
## Resilient storage with Longhorn
In Kubernetes world, the most difficult while essential part is probably the storage. It's not easy to find a solution that combine resiliency, scalability and performance.
{{< alert >}}
If you are not familiar with Kubernetes storage, you must at least be aware of pros and cons of `RWO` and `RWX` volumes when creating `PVC`.
In general `RWO` is more performant, but only one pod can mount it, while `RWX` is slower, but allow sharing between multiple pods.
`RWO` is a single node volume, and `RWX` is a shared volume between multiple nodes.
{{</ alert >}}
`K3s` come with a built-in `local-path` provisioner, which is the most performant `RWO` solution by directly using local NVMe SSD. But it's not resilient neither scalable. I think it's a good solution for what you consider at no critical data.
A dedicated NFS server is a good `RWX` solution, by using [this provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner). It allows scalability and resiliency with [GlusterFS](https://www.gluster.org/). But it stays a single point of failure in case of network problems, and give of course low IOPs. It's also a separate server to maintain.
For Hetzner, the easiest `RWO` solution is to use the [official CSI](https://github.com/hetznercloud/csi-driver) for automatic block volumes mounting. It's far more performant than NFS (but still less than local SSD), but there is no resiliency neither scalability. It's really easy to go with and very resource efficient for the cluster, note that multiple pods can [reference same volume](https://github.com/hetznercloud/csi-driver/issues/146) which allow reusability without wasting 10 GB each time.
As a more advanced solution storage, [Longhorn](https://longhorn.io/) seems to get some traction by combining most requirements with nice UI, with the price of high resource usage inside cluster. Moreover, it offers integrated backup solution with snapshots and remote S3, which avoid us to have to manage a dedicated backup solution like [velero](https://velero.io/) and some annotations for each pod volumes to be backup.
### Storage node pool
When it comes storage management, it's generally recommended having a dedicated node pool for it for better scalability.
{{< mermaid >}}
flowchart TB
subgraph worker-01
app-01([My App replica 1])
end
subgraph worker-02
app-02([My App replica 2])
end
subgraph worker-03
app-03([My App replica 3])
end
overlay(Overlay network)
worker-01 --> overlay
worker-02 --> overlay
worker-03 --> overlay
overlay --> storage-01
overlay --> storage-02
subgraph storage-01
longhorn-01[(Longhorn<br>volume)]
end
subgraph storage-02
longhorn-02[(Longhorn<br>volume)]
end
streaming(Data replication)
storage-01 --> streaming
storage-02 --> streaming
{{</ mermaid >}}
Let's get back to our 1st Hcloud Terraform Project, and add a new node pool for storage:
{{< highlight file="kube.tf" >}}
```tf
module "hcloud_kube" {
//...
agent_nodepools = [
//...
{
name = "storage"
server_type = "cx21"
location = "nbg1"
count = 2
private_interface = "ens10"
labels = [
"node.kubernetes.io/server-usage=storage"
]
taints = [
"node-role.kubernetes.io/storage:NoSchedule"
]
}
]
}
```
{{< /highlight >}}
Be sure to have labels and taints correctly set, as we'll use them later for Longhorn installation. This node pool will be dedicated for storage, so the tainted label will prevent any other pod workload to be scheduled on it.
After `terraform apply`, check that new storage nodes are ready with `kgno`. Now we'll also apply a configurable dedicated block volume on each node for more flexible space management.
{{< highlight file="kube.tf" >}}
```tf
module "hcloud_kube" {
//...
agent_nodepools = [
//...
{
name = "storage"
//...
volume_size = 10
}
]
}
```
{{< /highlight >}}
SSH to both storage nodes to check if a 20GB volume is correctly mounted by `df -h` command. It should be like :
```txt
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 38G 4,2G 32G 12% /
...
/dev/sdb 20G 24K 19,5G 1% /mnt/HC_Volume_XXXXXXXX
```
The volume is of course automatically mounted on each node reboot, it's done via `fstab`.
{{< alert >}}
Note as if you set volume in same time as node pool creation, Hetzner doesn't seem to automatically mount the volume. So it's preferable to create the node pool first, then add the volume as soon as the node in ready state. You can always delete and recreate volume by commenting then uncommenting `volume_size` variable, which will force a remount properly.
{{</ alert >}}
### Longhorn installation
Return to the 2nd Kubernetes terraform project, and add Longhorn installation:
{{< highlight file="longhorn.tf" >}}
```tf
```
{{< /highlight >}}
## PostgreSQL with replication
@ -20,6 +147,6 @@ This is the **Part IV** of more global topic tutorial. [Back to first part]({{<
## Backups (dumps + longhorn snapshots)
## 2nd check ✅
## 3th check ✅
Our cluster is now perfectly securely accessible from outside with minimal setup needed for any new apps. Persistence is insured by Longhorn with replicated storage and Minio for S3 needs. The next important part is now to have a [working database for real world apps]({{< ref "/posts/13-build-your-own-kubernetes-cluster-part-4" >}}).
Persistence is now insured by Longhorn as a resilient storage and a DB replicated cluster. It's now time to test some [real world apps]({{< ref "/posts/13-build-your-own-kubernetes-cluster-part-4" >}}) with a proper CD solution.