Alertmanager + Slack

Deploying Alertmanager in Kubernetes

Alertmanager is a utility service deployed alongside Prometheus to help route, group, and prioritize alerts. Alertmanager does not trigger alerts - alerting rules are set in Prometheus itself, Alertmanager only takes care of the lifecycle of the alert after it has been triggered (you can think of it as Zapier but for Prometheus alerts).

Alertmanager is part of the Prometheus stack, the most popular open-source monitoring solution for Kubernetes, therefore deploying and configuring it is pretty straightforward.


Autometrics-instrumented application with SLOs

Autometrics enables you to set up smarter alerts using the Service Level Objectives (SLO) approach. You can read more about SLOs in Autometrics here and follow the guides for SLO-instrumentation under each supported language:


Create a monitoring namespace

It is a good practice to keep all monitoring resources in a separate namespace. To create a namespace run the following command (if you already have a namespace for monitoring tools, you can skip this step):

kubectl create namespace monitoring

Deploy Alertmanager to Kubernetes

Run the following command to install Alertmanager using the Helm chart. If you've already installed Prometheus using Helm, you can skip this step.

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

This will install Alertmanager in your cluster along with Prometheus, Node Exporter, and a few other components. As these components are installed together they are configured to work with each other out of the box.

You can now configure your Alertmanager to group, organize, and send alerts to whichever channel you're using for incident response. See Alertmanager documentation (opens in a new tab) for details.

Add the Autometrics Slack App

Autometrics Slack App is a simple open-source, self-hosted Slack service for Autometrics SLO alerts. It integrates with the Alertmanager and adds more context for each triggered message: the Slack app pulls recent relevant Prometheus data and renders it as an image of the graph on Slack.

Autometrics Slack app is open-source and available as a Docker image (opens in a new tab). Here's a quick guide how to add it and configure it with your Prometheus and Alertmanager setup.

Create a new Slack application

Go to (opens in a new tab), select your workspace, and create an application from manifest.

This is a basic Slack application manifest with limited permissions to post chat messages on channels that the application gets added to. You can find the manifest below:

  major_version: 1
  minor_version: 0
  name: Autometrics
  description: This bot provides a way for our Autometrics slack-app to communicate with the Slack API.
  long_description: |
    This bot provides a way for our Autometrics slack-app to communicate with
    the Slack API. The autometrics slack-app will post messages to a specified
    channel whenever a alert occurs in your alertmanager (note this requires
    modifications in your alertmanager configuration).
  background_color: "#00FFAA"
  org_deploy_enabled: false
  socket_mode_enabled: false
  token_rotation_enabled: false
    home_tab_enabled: false
    messages_tab_enabled: false
    display_name: Autometrics
    always_online: false
      - chat:write

Install your new Slack application into your workspace in the Install your app section and grab the Bot User OAuth Token, starts with xoxb-***

Deploy the Slack application

You can deploy the Slack application as a Kubernetes deployment using the Docker image. The Slack application requires some configuration passed in as environment variables and needs access to the Prometheus, Alertmanager and Slack API.

Here are the necessary environment variables:

LISTEN_HOSTMake sure this address allows for remote connections
BASE_URLThe URL that the Slack should be accessible at (for Slack API)
SLACK_CHANNELWhich Slack channel the app should post alerts to (make sure the application is invited to it!)
SLACK_BOT_TOKENThe xoxb-*** value we grabbed earlier when creating the application
STORAGE_DIRThe directory where the generated alert images should be stored. Make sure it's persistent storage
DB_CONNECTION_STRINGThe sqlite connection string for historic alert storage. Make sure it's persistent

Here's an example Kubernetes deployment manifest with the above configuration in place:

apiVersion: apps/v1
kind: Deployment
  name: slack-app
    app: slack-app
      app: slack-app
        app: slack-app
        - name: slack-app
          image: "autometrics/slack-app:latest"
          imagePullPolicy: Always
          resources: {}
            - name: RUST_LOG
              value: "info"
            - name: LOG_JSON
              value: "true"
            - name: LISTEN_HOST
            - name: PORT
              value: "3031"
            - name: BASE_URL
              value: "" # NOTE: this url should be accessible to Slack API
            - name: EXPLORER_URL
            - name: SLACK_CHANNEL
              value: "test-slack-app"
            - name: SLACK_BOT_TOKEN
              value: xoxb-000000000-000000000-000000000
            - name: PROMETHEUS_URL
              value: http://prometheus:9090
            - name: STORAGE_DIR
              value: /data/
            - name: DB_CONNECTION_STRING
              value: "sqlite:///data/slack-app.db?mode=rwc"
            - containerPort: 3031
            - name: storage-volume
              mountPath: /data
        - name: storage-volume
            claimName: slack-app

Update Alertmanager configuration

Finally we need to update the Alertmanager so that it sends a Webhook request whenever an alert is triggered to our newly deployed Slack application:

- name: default-receiver
  - send_resolved: true
    url: 'http://slack-app:3031/api/alerts'
  receiver: default-receiver