Deploying Alertmanager in Kubernetes

Alertmanager is a utility service deployed alongside Prometheus to help route, group, and prioritize alerts. Alertmanager does not trigger alerts - alerting rules are set in Prometheus itself, Alertmanager only takes care of the lifecycle of the alert after it has been triggered (you can think of it as Zapier but for Prometheus alerts).

Alertmanager is part of the Prometheus stack, the most popular open-source monitoring solution for Kubernetes, therefore deploying and configuring it is pretty straightforward.

Requirements

Kubernetes cluster running with kubectl configured to access it
Helm (opens in a new tab) installed

Autometrics-instrumented application with SLOs

Autometrics enables you to set up smarter alerts using the Service Level Objectives (SLO) approach. You can read more about SLOs in Autometrics here and follow the guides for SLO-instrumentation under each supported language:

Instructions

Create a `monitoring` namespace

It is a good practice to keep all monitoring resources in a separate namespace. To create a namespace run the following command (if you already have a namespace for monitoring tools, you can skip this step):

kubectl create namespace monitoring

Deploy Alertmanager to Kubernetes

Run the following command to install Alertmanager using the Helm chart. If you've already installed Prometheus using Helm, you can skip this step.

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

This will install Alertmanager in your cluster along with Prometheus, Node Exporter, and a few other components. As these components are installed together they are configured to work with each other out of the box.

You can now configure your Alertmanager to group, organize, and send alerts to whichever channel you're using for incident response. See Alertmanager documentation (opens in a new tab) for details.

Add the Autometrics Slack App

Autometrics Slack App is a simple open-source, self-hosted Slack service for Autometrics SLO alerts. It integrates with the Alertmanager and adds more context for each triggered message: the Slack app pulls recent relevant Prometheus data and renders it as an image of the graph on Slack.

Autometrics Slack app is open-source and available as a Docker image (opens in a new tab). Here's a quick guide how to add it and configure it with your Prometheus and Alertmanager setup.

Create a new Slack application

Go to https://api.slack.com/apps/ (opens in a new tab), select your workspace, and create an application from manifest.

This is a basic Slack application manifest with limited permissions to post chat messages on channels that the application gets added to. You can find the manifest below:

manifest.yml

_metadata:
  major_version: 1
  minor_version: 0
display_information:
  name: Autometrics
  description: This bot provides a way for our Autometrics slack-app to communicate with the Slack API.
  long_description: |
    This bot provides a way for our Autometrics slack-app to communicate with
    the Slack API. The autometrics slack-app will post messages to a specified
    channel whenever a alert occurs in your alertmanager (note this requires
    modifications in your alertmanager configuration).
  background_color: "#00FFAA"
settings:
  org_deploy_enabled: false
  socket_mode_enabled: false
  token_rotation_enabled: false
features:
  app_home:
    home_tab_enabled: false
    messages_tab_enabled: false
  bot_user:
    display_name: Autometrics
    always_online: false
oauth_config:
  scopes:
    bot:
      - chat:write

Install your new Slack application into your workspace in the Install your app section and grab the Bot User OAuth Token, starts with xoxb-***

Deploy the Slack application

You can deploy the Slack application as a Kubernetes deployment using the Docker image. The Slack application requires some configuration passed in as environment variables and needs access to the Prometheus, Alertmanager and Slack API.

Here are the necessary environment variables:

Variable	Description
`LISTEN_HOST`	Make sure this address allows for remote connections
`BASE_URL`	The URL that the Slack should be accessible at (for Slack API)
`SLACK_CHANNEL`	Which Slack channel the app should post alerts to (make sure the application is invited to it!)
`SLACK_BOT_TOKEN`	The `xoxb-***` value we grabbed earlier when creating the application
`STORAGE_DIR`	The directory where the generated alert images should be stored. Make sure it's persistent storage
`DB_CONNECTION_STRING`	The sqlite connection string for historic alert storage. Make sure it's persistent

Here's an example Kubernetes deployment manifest with the above configuration in place:

deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: slack-app
  labels:
    app: slack-app
spec:
  selector:
    matchLabels:
      app: slack-app
  template:
    metadata:
      labels:
        app: slack-app
    spec:
      containers:
        - name: slack-app
          image: "autometrics/slack-app:latest"
          imagePullPolicy: Always
          resources: {}
          env:
            - name: RUST_LOG
              value: "info"
            - name: LOG_JSON
              value: "true"
            - name: LISTEN_HOST
              value: 0.0.0.0
            - name: PORT
              value: "3031"
            - name: BASE_URL
              value: "https://slack-app.example.com" # NOTE: this url should be accessible to Slack API
            - name: EXPLORER_URL
              value: https://explorer.autometrics.dev
            - name: SLACK_CHANNEL
              value: "test-slack-app"
            - name: SLACK_BOT_TOKEN
              value: xoxb-000000000-000000000-000000000
            - name: PROMETHEUS_URL
              value: http://prometheus:9090
            - name: STORAGE_DIR
              value: /data/
            - name: DB_CONNECTION_STRING
              value: "sqlite:///data/slack-app.db?mode=rwc"
          ports:
            - containerPort: 3031
          volumeMounts:
            - name: storage-volume
              mountPath: /data
      volumes:
        - name: storage-volume
          persistentVolumeClaim:
            claimName: slack-app

Update Alertmanager configuration

Finally we need to update the Alertmanager so that it sends a Webhook request whenever an alert is triggered to our newly deployed Slack application:

alertmanager.yml

global:
receivers:
- name: default-receiver
  webhook_configs:
  - send_resolved: true
    url: 'http://slack-app:3031/api/alerts'
route:
  receiver: default-receiver

Docker OTel Collector