Monitoring and Logging installation

This document contains instructions for installing the Monitoring and Logging components in the Kubernetes cluster.

Add the Monitoring node to the Kubernetes cluster

To install the Monitoring and Logging components in the Kubernetes cluster, use a separate host, VM3 (Virtual Machine 3), equipped with a local disk to store both the monitoring and logging data.

Monitoring and Logging consist of the following key components:

Grafana
InfluxDB
Prometheus
Loki
Fluent Bit

To add the Monitoring node to the Kubernetes cluster, prepare the host and install Ubuntu Server 22.04 LTS.

Prepare the host

Prepare the host VM3 (Virtual Machine 3) that matches the following requirements:

8 vCPU
32GB vRAM
vDisk1 100GB (sda), the system disk
vDisk2 500GB (sdb), will be used to store monitoring and logging data
1 vNIC, IP address
OS: Ubuntu Server 22.04 LTS
A hostname example: k8s-monitoring-xx (xx: 01, 02 … )

Install Ubuntu Server 22.04 LTS

TTo install Ubuntu Server 22.04 LTS, follow these steps:

Install the latest updates for Ubuntu Server 22.04 LTS:

Bash

sudo apt update && sudo apt upgrade -y

Enable SSH key-based access for the admin user by copying SSH keys into ~/.ssh/authorized_keys.
Enable sudo access with no password for the current admin user:

Bash

sudo bash -c "cat << EOF >> /etc/sudoers

$USER ALL=(ALL) NOPASSWD:ALL
EOF"

Reboot the server.

Bash

sudo reboot

Add the host to the Kubernetes cluster

All operations will be executed on VM1, unless an alternative is explicitly designated.

On VM3, clone repository. Replace {TAG_NAME} with one of available tags (to list all available tags, use git tag):

Bash

git clone https://github.com/aipix-dev/aipix-deploy.git
cd ./aipix-deploy
git checkout {TAG_NAME}
cd kubernetes/k8s-onprem/

Install the Kubernetes base components:

Bash

./install_kube_base.sh

On VM1, get the registration token and URL:

Bash

kubeadm token create --print-join-command

Bash

###Examle token:
kubeadm join 192.168.205.164:6443 --token 3yeqrm.abnp3yof8vivcbge --discovery-token-ca-cert-hash sha256:b58cfd679a3bb49f444dfe4869fad5e19f4fba87f1d6ae5f20da06c60f51684e

Copy the token and apply it with sudo on VM3:

Bash

###Example:
sudo kubeadm join 192.168.205.164:6443 --token 3yeqrm.abnp3yof8vivcbge --discovery-token-ca-cert-hash sha256:b58cfd679a3bb49f444dfe4869fad5e19f4fba87f1d6ae5f20da06c60f51684e

On VM1, check if the new node is added with the Ready status:

Bash

kubectl get nodes

Bash

###Example output:
NAME                   STATUS   ROLES           AGE     VERSION
k8s-monitoring-01      Ready    <none>          3m28s   v1.28.2

On VM1, label and taint the new node to be used only by Monitoring (replace the k8s-monitoring-01 node name with your node name):

Bash

##Example:
kubectl taint nodes k8s-monitoring-01 monitoring:NoSchedule
kubectl label nodes k8s-monitoring-01 monitoring=true

Prepare a local storage by running the following script on VM3:

Bash

cd ~/aipix-deploy/kubernetes/k8s-onprem
./prepare_local_storage.sh 1 7 sdb make_fs

Install the Monitoring and Logging components

All operations will be executed on VM1, unless an alternative is explicitly designated.

To install the Monitoring and Logging components, follow these steps:

On VM1, go to the directory:

Bash

cd ~/aipix-deploy/kubernetes

Copy sources.sh.sample to sources.sh if it was not copied before, and check the configurations of the monitoring section.
For proper Analytics monitoring, it is required for the variable MONITORING to be set to yes during the Analytics installation. Update the Analytics installation if necessary.

Bash

vim ./sources.sh

Bash

#Monitoring parameters
export MONITORING=no                                 #If monitoring to be deployed ("yes" or "no")
export NS_MONITORING=monitoring                      #k8s namespace for monitoring
export INFLUX_USR=admin                              #define influxdb admin user
export INFLUX_PSW=0hmSYYaRci6yJblARc6aHbHZ4YelTXTo   #define influxdb admin userpassword (use letters and numbers only)
export INFLUX_TOKEN=2pORp9tDo40Lm32oGUKFLL8r1UuNbgUT #define influxdb API token (use letters and numbers only)
export TELEGRAM_BOT_TOKEN=temporary-token            # define a Telegram bot token for notifications
export TELEGRAM_CHAT_ID=temporary-chat-id            # define a Telegram chat iD for notifications

Prepare the configuration files for Monitoring:

Bash

./configure-monitoring.sh

Check the configuration files in the../monitoring/ folder and make changes if required. To monitor S3 MinIO deployment, adjust bearer_token: in job_name: minio-job , job_name: minio-job-bucket in the prometheus-config-map.yaml file.
To get the tokens, run the following commands (in this example, local – is the MinIO S3 storage alias name):

Bash

mc admin prometheus generate local (local, minio-1 or minio-2)

Bash

mc admin prometheus generate local bucket

Adjust bearer_token: (or the entire block of MinIO configurations) in the prometheus-config-map.yaml file (vim ../monitoring/prometheus-config-map.yaml or vim ../monitoring/prometheus-values.yaml ):

Bash

  - job_name: minio-job-cluster
    bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoibmF2ZWtzb2Z0IiwiZXhwIjo0ODUwMzk2OTA3fQ.WWCLGWb-usnTqR5aWGDwRpSlgl0VfDvthSSWYxhf3X1UNXCRWoGIZz386Q3KIShzAHipamIoFyf5oR9YCebobg
    metrics_path: /minio/v2/metrics/cluster
    scheme: http
    static_configs:
      - targets: [minio.minio-single.svc:9000]
  - job_name: minio-job-bucket
    bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoibmF2ZWtzb2Z0IiwiZXhwIjo0ODUwNTIwMjkyfQ.acVDRmB-kLSFCkzIIZAinzTRguHYYKZeM0ebwHHKfSDhjq3xYb4YaNwfHrJKoI4E3z5fTGD7sal4QwF5nxdn-w
    metrics_path: /minio/v2/metrics/bucket
    scheme: http
    static_configs:
      - targets: [minio.minio-single.svc:9000]

Install Monitoring and Logging by running the following script:

Bash

./deploy-monitoring.sh

When the installation is complete, you can connect to Grafana and import dashboards from the dashboard collection located in the ~/aipix-deploy/monitoring/grafana-dashboards/ folder of the current repository, provided that automatic dashboard provisioning is not set.

At the end of the script, you’ll get URLs for different components.