A client wished to setup and configure an Alert Manager in Kubernetes (AWS EKS) which sends alerts from Prometheus to Microsoft (MS) Teams channel through AWS SNS (Simple Notification Service). The Alert Manager supports and provides configuration for AWS SNS; however, we chose to use prometheus-msteams helm chart.
Prometheus is installed in the EKS cluster using the helm chart kube prometheus stack.
What is Helm chart?
Helm chart is a package manager for Kubernetes that bundles or packages the application into charts. A chart is a collection of files that can be described as a set of Kubernetes resources.
What is Prometheus MS Teams?
Prometheus msteams is a server that receives an alert message as a POST message from the Prometheus Alert Manager and sends it to the Microsoft Teams channel using the incoming webhook URL.
Create incoming webhook URL in MS Teams channel
click on more options and then connectors as shown below from within your respective MS Teams channel where you need to get alert messages
Search and select Incoming Webhook and click on Add. Then click on Configure, enter a name and upload an image for your webhook if needed.
Copy and save the webhook URL shown in the window.
Install kube-prometheus-stack Helm chart in the EKS cluster
(You can skip this step if you have installed this helm chart or any other prometheus and alert manager helm chart)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack
(Note – In our case we have used RELEASE_NAME as prometheus-stack)
If you want to install helm chart using custom values, then edit the values.yaml file of the helm chart and then use the command,
helm install –f values.yaml [RELEASE_NAME] prometheus-community/kube-prometheus-stack OR helm upgrade --install -f values.yaml [RELEASE_NAME] prometheus-community/kube-prometheus-stack
This helm chart includes Prometheus operator, Prometheus, alert manager, Prometheus node-exporter, Kube-state-metrics, Prometheus adapter for Kubernetes Metrics APIs and Grafana.
Install prometheus-ms-teams Helm chart in the EKS cluster
helm repo add prometheus-msteams
Create a values.yaml file and pass the following values to the file.
(Note: Mention the value of the alert as your unique webhook URL and also the correct kube-prometheus-stack helm chart release name. In our case we use prometheus-stack)
Then install the helm chart using,
helm upgrade --install prometheus-msteams \ --namespace default -f values.yaml prometheus-msteams/prometheus-msteams
Create a values.yaml file and pass the following values to the file.
Create an alert manager configuration file and add that configuration file in Kubernetes secrets
Where prometheus-msteams from webhook_configs url is the Kubernetes service name of Prometheus MS Teams. Name the configuration file alertmanager.yaml.
Create a Kubernetes secret that contains this alert manager configuration,
kubectl create secret generic myalertmanager --from-file=alertmanager.yaml=alertmanager.yaml
You can check the secret content using,
kubectl get secret myalertmanager -o yaml
Copy the value of alertmanager.yaml as shown above in the image and enter the following command to check whether correct alert manager configuration is passed to the Kubernetes secret,
echo "<value-of-alertmanager.yaml>” | base64 -d
Mention this secret in the alertmanager section of the kube prometheus stack helm chart values.yaml
Upgrade the helm chart kube-prometheus-stack
helm upgrade -f values.yaml prometheus-stack kube-prometheus-stack
Once updated, check if alerts are as shown (as seen in the images below)
About the author
Shubham Lonkar
Shubham is a DevOps engineer at Excellarate for 4 years. He has expertise in tools and technologies such as Docker, Kubernetes, terraform, Jenkins, Gitlab CICD, Bitbucket CICD. Shubham holds a bachelor degree in Computer Engineering from the Savitribai Phule Pune University.
