okami101/blog
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 46 Wiki Activity

proofreading

Browse Source
This commit is contained in:
Adrien Beaudouin
2022-02-27 21:22:08 +01:00
parent 4e574de61b
commit ac96fac378
4 changed files with 130 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
content/posts/04-build-your-own-docker-swarm-cluster-part-3/index.md
View File

@ -398,6 +398,8 @@ It's finally time to test our new cluster environment by testing some stacks thr
Create a new `diun` stack through Portainer and set following content :
{{< highlight host="stack" file="diun">}}
```yml
version: "3.2"
@ -425,6 +427,8 @@ services:
- node.role == manager
```
{{< /highlight >}}
{{< tabs >}}
{{< tab tabName="volumes" >}}

2
content/posts/05-build-your-own-docker-swarm-cluster-part-4/index.md
View File

@ -260,7 +260,7 @@ services:
- traefik.http.services.matomo.loadbalancer.server.port=80
placement:
constraints:
- node.role == manager
- node.labels.environment == production
networks:
traefik_public:

61
content/posts/06-build-your-own-docker-swarm-cluster-part-5/index.md
View File

@ -38,7 +38,9 @@ It's equivalent of doing :
docker node update --label-add prometheus.data=true manager-01
```
Then create a config file at `/etc/prometheus/prometheus.yml` in `manager-01` node :
Then create following config file :
{{< highlight host="manager-01" file="/etc/prometheus/prometheus.yml" >}}
```yml
global:
@ -52,8 +54,11 @@ scrape_configs:
- job_name: "traefik"
static_configs:
- targets: ["traefik_traefik:8080"]
```
{{< /highlight >}}
It consists on 2 scrapes job, use `targets` in order to indicate to Prometheus the `/metrics` endpoint locations. I configure `5s` as interval, that means Prometheus will scrape `/metrics` endpoints every 5 seconds.
Finally create a `prometheus` stack in Portainer :
@ -96,10 +101,10 @@ volumes:
The `private` network will serve us later for exporters. Next config are useful in order to control the DB usage, as metrics can go up very quickly :
| argument | description |
| --------------------------- | --------------------------- |
| storage.tsdb.retention.size | The max DB size |
| storage.tsdb.retention.time | The max data retention date |
| argument | description |
| ----------------------------- | --------------------------- |
| `storage.tsdb.retention.size` | The max DB size |
| `storage.tsdb.retention.time` | The max data retention date |
Deploy it and <https://prometheus.sw.dockerswarm.rocks> should be available after few seconds. Use same traefik credentials for login.
@ -115,8 +120,8 @@ In *Status > Targets*, you should have 2 endpoints enabled, which correspond to
We have the monitor brain, new it's time to have some more relevant metrics data from all containers as well as docker nodes. It's doable thanks to exporters :
* `cAdvisor` from Google which scrape metrics of all running containers
* `Node exporter` for more global cluster evaluation
* **cAdvisor** from Google which scrape metrics of all running containers
* **Node exporter** for more global cluster evaluation
Before edit above stack, we need to make a specific docker entrypoint for node exporter that will help us to fetch the original hostname of the docker host machine name. This is because we run node exporter as docker container, which have no clue of docker hostname.
@ -124,6 +129,8 @@ Besides this node exporter (like cAdvisor) work as an agent which must be deploy
Go to *Configs* menu inside Portainer and add a `node_exporter_entrypoint` config file with next content :
{{< highlight host="config" file="node_exporter_entrypoint" >}}
```sh
#!/bin/sh -e
@ -135,6 +142,8 @@ set -- /bin/node_exporter "$@"
exec "$@"
```
{{< /highlight >}}
[![Portainer configs](portainer-configs.png)](portainer-configs.png)
It will take the node hostname and create an exploitable data metric for prometheus.
@ -186,7 +195,9 @@ configs:
external: true
```
Finally, add the 2 next jobs on `/etc/prometheus/prometheus.yml` :
Finally, add the 2 next jobs in previous Prometheus config file :
{{< highlight host="manager-01" file="/etc/prometheus/prometheus.yml" >}}
```yml
#...
@ -206,6 +217,8 @@ Finally, add the 2 next jobs on `/etc/prometheus/prometheus.yml` :
#...
```
{{< /highlight >}}
The `tasks.*` is a specific DNS from specific to Docker Swarm which allows multiple communication at once when using *global* mode, similarly as `tcp://tasks.agent:9001` for Portainer.
You need to restart Prometheus service in order to apply above config.
@ -226,12 +239,16 @@ Okay so now we have plenty metrics from our cluster and containers, but Promethe
Before install Grafana, let's quickly install a powerful key-value database cache on `data-01` :
{{< highlight host="data-01" >}}
```sh
sudo add-apt-repository ppa:redislabs/redis
sudo apt install -y redis-server
sudo systemctl enable redis-server.service
```
{{< /highlight >}}
### Grafana install 💽
As always, it's just a Swarm stack to deploy ! Like [N8N]({{< ref "/posts/05-build-your-own-docker-swarm-cluster-part-4#n8n-over-postgresql" >}}), we'll use a proper real production database and production cache.
@ -240,11 +257,15 @@ First connect to pgAdmin and create new grafana user and database. Don't forget
Create storage folder with :
{{< highlight host="manager-01" >}}
```sh
sudo mkdir /mnt/storage-pool/grafana
sudo chown -R 472:472 /mnt/storage-pool/grafana
```
{{< /highlight >}}
Next create new following `grafana` stack :
```yml
@ -308,6 +329,8 @@ We have done for the cluster metrics part but what about the external `data-01`
For node exporter, we have no other choice to install it locally as a service binary, so we must go through old fashion install.
{{< highlight host="data-01" >}}
```sh
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xzf node_exporter-1.3.1.linux-amd64.tar.gz
@ -316,9 +339,13 @@ sudo mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/
rm -r node_exporter-1.3.1.linux-amd64/
```
Create a new systemd file service `/etc/systemd/system/node-exporter.service` :
{{< /highlight >}}
```conf
Create a new systemd file service :
{{< highlight host="data-01" file="/etc/systemd/system/node-exporter.service" >}}
```txt
[Unit]
Description=Node Exporter
@ -330,14 +357,20 @@ ExecStart=/usr/local/bin/node_exporter
WantedBy=default.target
```
{{< /highlight >}}
Then enable the service and check status :
{{< highlight host="data-01" >}}
```sh
sudo systemctl enable node-exporter.service
sudo systemctl start node-exporter.service
sudo systemctl status node-exporter.service
```
{{< /highlight >}}
### Exporter for databases
For MySQL, we need to create a specific `exporter` user. Do `sudo mysql` and execute following SQL (replace *** by your password) :
@ -387,8 +420,11 @@ Set proper `MYSQL_PASSWORD` and `POSTGRES_PASSWORD` environment variables and de
Expand the prometheus config with 3 new jobs :
{{< highlight host="manager-01" file="/etc/prometheus/prometheus.yml" >}}
```yml
- job_name: "node-exporter-data-01"
#...
- job_name: "node-exporter-data-01"
static_configs:
- targets: ["data-01:9100"]
@ -399,8 +435,11 @@ Expand the prometheus config with 3 new jobs :
- job_name: "postgres-exporter-data-01"
static_configs:
- targets: ["postgres-exporter:9187"]
#...
```
{{< /highlight >}}
Then restart Prometheus service and go back to targets to check you have all new `data-01` endpoints.
[![Prometheus targets data](prometheus-targets-data.png)](prometheus-targets-data.png)

85
content/posts/07-build-your-own-docker-swarm-cluster-part-6/index.md
View File

@ -39,6 +39,8 @@ The mains exporters are :
First, let's install the main Loki service on `data-01` (be sure to have unzip with `sudo apt install -y unzip`) :
{{< highlight host="data-01" >}}
```sh
curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.2/loki-linux-amd64.zip"
unzip "loki-linux-amd64.zip"
@ -46,8 +48,12 @@ chmod a+x "loki-linux-amd64"
sudo mv loki-linux-amd64 /usr/local/bin/loki
```
{{< /highlight >}}
Prepare the config file :
{{< highlight host="data-01" >}}
```sh
wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml
sudo mkdir /etc/loki
@ -56,9 +62,13 @@ sudo mkdir /var/lib/loki
sudo chown swarm:swarm /var/lib/loki
```
{{< /highlight >}}
Edit `/etc/loki/loki-local-config.yaml` and change `/tmp/loki` by `/var/lib/loki`.
Then prepare the service `/etc/systemd/system/loki.service` :
Then prepare the service :
{{< highlight host="data-01" file="/etc/systemd/system/loki.service" >}}
```conf
[Unit]
@ -74,20 +84,28 @@ ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki-local-config.yaml
WantedBy=multi-user.target
```
{{< /highlight >}}
Finally, start the service :
{{< highlight host="data-01" >}}
```sh
sudo systemctl enable loki.service
sudo systemctl start loki.service
sudo systemctl status loki.service
```
{{< /highlight >}}
It's running !
### Data logs with Promtail
It's time to feed the Loki database with Promtail. First, let's install the main service, always in `data-01` (we don't need it on docker hosts) :
{{< highlight host="data-01" >}}
```sh
curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.2/promtail-linux-amd64.zip"
unzip "promtail-linux-amd64.zip"
@ -95,7 +113,11 @@ chmod a+x "promtail-linux-amd64"
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
```
Create `/etc/loki/promtail-local-config.yaml` :
{{< /highlight >}}
Create following file :
{{< highlight host="data-01" file="/etc/loki/promtail-local-config.yaml" >}}
```yml
server:
@ -123,11 +145,16 @@ scrape_configs:
job: postgresql-logs
host: data-01
__path__: /var/log/postgresql/*log
```
{{< /highlight >}}
The above config is pretty itself explanatory. We declare the URL of Loki rest API endpoint, and a list of jobs which consist of simple regex where to tail log files. The `positions.yaml` avoid duplications by keeping the last line where the service stopped for each log file.
Then prepare the service `/etc/systemd/system/promtail.service` :
Then prepare the service :
{{< highlight host="data-01" file="/etc/systemd/system/promtail.service" >}}
```conf
[Unit]
@ -142,14 +169,20 @@ ExecStart=/usr/local/bin/promtail -config.file=/etc/loki/promtail-local-config.y
WantedBy=multi-user.target
```
{{< /highlight >}}
Finally, start the service :
{{< highlight host="data-01" >}}
```sh
sudo systemctl enable promtail.service
sudo systemctl start promtail.service
sudo systemctl status promtail.service
```
{{< /highlight >}}
Recheck status after few seconds to confirm local var logs have been pushed successfully to Loki. Check `sudo cat /tmp/positions.yaml` for current tail status.
{{< alert >}}
@ -158,7 +191,7 @@ You can eventually repeat all this Promtail install procedure for each Docker ho
### Docker hosts
Now we need to push all container logs to Loki. The official [Docker driver](https://grafana.com/docs/loki/latest/clients/docker-driver) is a nice way to do it for perfect integration.
Now we need to push all container logs to Loki. The official [Docker driver](https://grafana.com/docs/loki/latest/clients/docker-driver) is a nice way to do it for perfect integration. Install it on all docker hosts :
```sh
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
@ -169,7 +202,9 @@ docker plugin ls
Now we have 2 options, reedit all active docker stack YAML description to use the Loki driver (boring), or downright consider it as default driver for all containers, which is relevant in our case, I think.
Create `/etc/docker/daemon.json` on each docker host with following content :
Create following file on each docker host with following content :
{{< highlight file="/etc/docker/daemon.json" >}}
```json
{
@ -181,6 +216,8 @@ Create `/etc/docker/daemon.json` on each docker host with following content :
}
```
{{< /highlight >}}
Then restart docker service `sudo service docker restart`.
And voilà, Loki is the default log driver for all containers. Note as you can still access your logs from Portainer.
@ -229,6 +266,8 @@ Elasticsearch is the recommended production choice for trace storage. I don't re
Let's install it on `data-01` :
{{< highlight host="data-01" >}}
```sh
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
@ -236,11 +275,15 @@ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee
sudo apt-get update && sudo apt-get install elasticsearch
```
{{< /highlight >}}
{{< alert >}}
Jaeger is not yet compatible with Elasticsearch 8...
{{< /alert >}}
Then allow remote network access add docker hosts by editing `/etc/elasticsearch/elasticsearch.yml` :
Then allow remote network access by adding docker hosts :
{{< highlight host="data-01" file="/etc/elasticsearch/elasticsearch.yml" >}}
```yml
#...
@ -250,24 +293,36 @@ discovery.seed_hosts: ["manager-01", "worker-01", "runner-01"]
#...
```
Before starting, let's calm down Java legendary memory consumption by creating `/etc/elasticsearch/jvm.options.d/hs.options` with following content :
{{< /highlight >}}
Before starting, let's calm down Java legendary memory consumption by creating following file :
{{< highlight host="data-01" file="/etc/elasticsearch/jvm.options.d/hs.options" >}}
```conf
-Xms512m
-Xmx512m
```
{{< /highlight >}}
Then start the service :
{{< highlight host="data-01" >}}
```sh
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
```
{{< /highlight >}}
Be sure that Elasticsearch is correctly responding from docker nodes by doing `curl http://data-01:9200`.
As a bonus, expand above `/etc/loki/promtail-local-config.yaml` by adding a new job :
As a bonus, expand above promtail config file by adding a new job :
{{< highlight host="data-01" file="/etc/loki/promtail-local-config.yaml" >}}
```yml
#...
@ -278,6 +333,8 @@ As a bonus, expand above `/etc/loki/promtail-local-config.yaml` by adding a new
#...
```
{{< /highlight >}}
Restart Promtail with `sudo service promtail restart`.
### Jaeger cluster installation
@ -346,7 +403,9 @@ It's time to inject some trace data. Be sure all above Jaeger services are start
### Traefik integration
Edit `/etc/traefik/traefik.yml` and add following `tracing` option :
Edit Traefik config file and add following `tracing` option :
{{< highlight host="manager-01" file="/etc/traefik/traefik.yml" >}}
```yml
#...
@ -357,7 +416,11 @@ tracing:
#...
```
Then edit original `traefik-stack.yml` file and add `traefik` service into `jaeger` network.
{{< /highlight >}}
Then edit original Traefik stack file and add `traefik` service into `jaeger` network.
{{< highlight host="manager-01" file="~/traefik-stack.yml" >}}
```yml
version: '3.2'
@ -376,6 +439,8 @@ networks:
external: true
```
{{< /highlight >}}
Then redeploy the stack by `docker stack deploy -c traefik-stack.yml traefik`. You'll probably need to reexport the `HASHED_PASSWORD` variable environment.
Go back to Traefik dashboard and ensure Jaeger is enabled in *Features* section. Traefik should now correctly send traces to Jaeger agent.