How to Monitor and Alert Bold Reports Using Grafana and Prometheus in Kubernetes

Published:

Updated:

This guide provides the steps to configure Grafana and Prometheus in a Kubernetes cluster. Monitoring the performance of your Kubernetes cluster is crucial for maintaining the health and efficiency of your applications. By connecting to the Kubernetes cluster you want to monitor, and setting up Grafana and Prometheus, you will be able to collect, visualize, and analyze performance metrics in real-time. This setup will enable you to effectively monitor, troubleshoot, and optimize your Kubernetes environment.

Configure Grafana and Prometheus:

Follow these steps to configure Grafana and Prometheus in the Kubernetes cluster:

Connect to the Kubernetes cluster that you want to monitor.

If Helm is not installed on your machine, you can install it using Chocolatey with the following command.

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((NewObjectSystem.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Add the Prometheus Helm repository using the following command.

 helm repo add stable https://charts.helm.sh/stable
 helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Update the Helm repository.
```
helm repo update
```

Add the Prometheus community Helm chart to Kubernetes.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

After running the above command to add the Prometheus and Grafana Helm charts, proceed to install the kube-prometheus-stack.

Install the stable Prometheus-community/kube-prometheus-stack.

helm install stable prometheus-community/kube-prometheus-stack

Once installed, you will see the following output:
```
kubectl get pods
```

Edit Grafana and Prometheus services for external access:

By default, the Prometheus and Grafana services are available within the cluster using ClusterIP. To access them externally, change the service type to either NodePort or LoadBalancer.
```
kubectl edit svc stable-kube-prometheus-sta-prometheus
```
Change the service type from ClusterIP to LoadBalancer or NodePort.
Edit the Grafana service in the same way as you did for Prometheus.
Run the command below to get the services.
```
Kubectl get svc
```
The LoadBalancer assigns an external IP to both the Grafana and Prometheus services.
Access Grafana and Prometheus using the provided IP address and port.
Add this Prometheus as a data source in Grafana.
Click Add data source.
Search for Prometheus and enter the external IP address and port number of your Prometheus service in the URL field.
Then, click Save & Test.
The data source is now configured in Grafana.

Dashboard Creation and Monitoring:

Grafana has a default dashboard for monitoring cluster performance.
Use the following Grafana dashboard ID for overall cluster monitoring.
Use the dashboard ID - 9873 and select Prometheus as the data source for the dashboard creation.
Refer here for more dashboards.

Configure Alerting:

Add a notification channel to receive alerts.
Enter your email address and click Save Contact Point.
Refer here for instructions on configuring SMTP in Grafana to receive emails.
Save and test the contact point.
Refer here for instructions on creating alerts in Grafana.
For example, to create a “Pods Down” alert rule, go to the Alerts section and click Add New Alert.
Search for a PromQL query(Prometheus Query Language) to find pods in your namespace.
Set the threshold to 1 and add the other details.
Click Save.
Find your alert rule under Alert Rules.
After creating the alert in Grafana, it will send an email notification to the specified address (e.g., Gmail) when the alert is triggered.