let's logs

content/posts/06-build-your-own-docker-swarm-cluster-part-5/index.md

@@ -293,6 +293,8 @@ For best show-case scenario of Grafana, let's import an [existing dashboard](htt
 
 First we need to add Prometheus as main metrics data source. Go to *Configuration > Data source* menu and click on *Add data source*. Select Prometheus and set the internal docker prometheus URL, which should be `http://prometheus:9090`.
 
+![Grafana prometheus datasource](grafana-prometheus-datasource.png)
+
 Then go to *Create > Import*, load `11939` as dashboard ID, and select Prometheus source and woha!
 
 ![Grafana home](grafana-docker-swarm-dashboard.png)

content/posts/07-build-your-own-docker-swarm-cluster-part-6/index.md

@@ -37,11 +37,10 @@ The mains exporters are :
 
 ## Logs with Loki 📄
 
-First, let's install the main Loki service :
+First, let's install the main Loki service on `data-01` (be sure to have unzip with `sudo apt install -y unzip`) :
 
 ```sh
 curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.2/loki-linux-amd64.zip"
-sudo apt install -y unzip
 unzip "loki-linux-amd64.zip"
 chmod a+x "loki-linux-amd64"
 sudo mv loki-linux-amd64 /usr/local/bin/loki
@@ -57,9 +56,9 @@ sudo mkdir /var/lib/loki
 sudo chown swarm:swarm /var/lib/loki
 ```
 
-> Change default /tmp/loki to /var/lib/loki
+Edit `/etc/loki/loki-local-config.yaml` and change `/tmp/loki` by `/var/lib/loki`.
 
-Then prepare the service `/etc/systemd/system/loki.service`
+Then prepare the service `/etc/systemd/system/loki.service` :
 
 ```conf
 [Unit]
@@ -75,26 +74,102 @@ ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki-local-config.yaml
 WantedBy=multi-user.target
 ```
 
-Finally, start the service
+Finally, start the service :
 
 ```sh
-sudo service loki start
-sudo service loki status
 sudo systemctl enable loki.service
+sudo systemctl start loki.service
 sudo systemctl status loki.service
 ```
 
+It's running !
+
 ### Data logs with Promtail
 
+It's time to feed the Loki database with Promtail. First, let's install the main service, always in `data-01` (we don't need it on docker hosts) :
+
+```sh
+curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.2/promtail-linux-amd64.zip"
+unzip "promtail-linux-amd64.zip"
+chmod a+x "promtail-linux-amd64"
+sudo mv promtail-linux-amd64 /usr/local/bin/promtail
+```
+
+Create `/etc/loki/promtail-local-config.yaml` :
+
+```yml
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://data-01:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: system
+    static_configs:
+      - labels:
+          job: varlogs
+          host: data-01
+          __path__: /var/log/*log
+      - labels:
+          job: mysql-logs
+          host: data-01
+          __path__: /var/log/mysql/*log
+      - labels:
+          job: postgresql-logs
+          host: data-01
+          __path__: /var/log/postgresql/*log
+```
+
+The above config is pretty itself explanatory. We declare the URL of Loki rest API endpoint, and a list of jobs which consist of simple regex where to tail log files. The `positions.yaml` avoid duplications by keeping the last line where the service stopped for each log file.
+
+Then prepare the service `/etc/systemd/system/promtail.service` :
+
+```conf
+[Unit]
+Description=Promtail
+After=network.target
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/promtail -config.file=/etc/loki/promtail-local-config.yaml
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Finally, start the service :
+
+```sh
+sudo systemctl enable promtail.service
+sudo systemctl start promtail.service
+sudo systemctl status promtail.service
+```
+
+Recheck status after few seconds to confirm local var logs have been pushed successfully to Loki. Check `sudo cat /tmp/positions.yaml` for current tail status.
+
+{{< alert >}}
+You can eventually repeat all this Promtail install procedure for each Docker host if you want to have system logs for all nodes.
+{{< /alert >}}
+
 ### Docker hosts
 
-<https://grafana.com/docs/loki/latest/clients/docker-driver/>
+Now we need to push all container logs to Loki. The official [Docker driver](https://grafana.com/docs/loki/latest/clients/docker-driver) is a nice way to do it for perfect integration.
 
 ```sh
 docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
+
+# ensure plugin is enabled
+docker plugin ls
 ```
 
-`/etc/docker/daemon.json`
+Now we have 2 options, reedit all active docker stack YAML description to use the Loki driver (boring), or downright consider it as default driver for all containers, which is relevant in our case, I think.
+
+Create `/etc/docker/daemon.json` on each docker host with following content :
 
 ```json
 {
@@ -106,8 +181,40 @@ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all
 }
 ```
 
+Then restart docker service `sudo service docker restart`.
+
+And voilà, Loki is the default log driver for all containers. Note as you can still access your logs from Portainer.
+
 ### Grafana explore and dashboard
 
+Now it's time to set up our central logs dashboard. First add *Loki* as a new data source inside Grafana, similarly to previous Prometheus. Set `http://data-01:3100` inside URL field and save it.
+
+![Grafana loki datasource](grafana-loki-datasource.png)
+
+Then create a new Dashboard. No need to import this time :
+
+1. Add a new panel
+2. Set logs as visualization type
+3. Select Loki in Data source
+4. Test some basic LogQL in Log browser in order to confirm all is working. Simply type `{` It should have full autocomplete. You should have plenty of access logs when using `{swarm_stack="traefik"}`
+
+![Grafana loki datasource](grafana-panel-editor.png)
+
+After this primary testing, let's use the power of Grafana with variables :
+
+1. Set `{swarm_stack="$stack"}"` in log browser
+2. Go to dashboard settings and enter the *Variables* section
+3. Create a `stack` variable, select Prometheus as *Data source*, and insert following value inside *Query* field : `label_values(container_last_seen, container_label_com_docker_stack_namespace)`
+4. It's a PromQL which fetch all detected docker stacks, click on *Update* to confirm the validity of *Preview of values* that will be show up
+
+![Grafana loki datasource](grafana-variables.png)
+
+1. Return to your panel editor. A new *stack* selector will appear in the top will all you to select the stack logs to show !
+2. Let's apply for saving the panel and test the selector. The Panel should reactive with the *stack* selector.
+3. Save the dashboard.
+
+![Grafana loki datasource](grafana-logs-dashboard.png)
+
 ## Tracing with Jaeger 🔍
 
 ### Traefik integration