Alertmanager + Slack

Deploying Alertmanager on Docker

Alertmanager is a utility service deployed alongside Prometheus to help route, group, and prioritize alerts. Alertmanager does not trigger alerts - alerting rules are set in Prometheus itself, Alertmanager only takes care of the lifecycle of the alert after it has been triggered (you can think of it as Zapier but for Prometheus alerts).

Alertmanager is available as a Docker image from the official Prometheus organization.



Make sure you have Prometheus running in the environment you're hoping to connect the Alertmanager to. See Deploying Prometheus to Docker.

Autometrics-instrumented application with SLOs

Autometrics enables you to set up smarter alerts using the Service Level Objectives (SLO) approach. You can read more about SLOs in Autometrics here and follow the guides for SLO-instrumentation under each supported language:


In the following guide we'll deploy a basic Alertmanager instance configured to work best with Autometrics alerts that will post a notification to Slack whenever there's a breach in the SLO we defined earlier with Autometrics.

Create a Slack application

First, we need to create a Slack application that will be used to send notifications to Slack. You can create a new Slack application here (opens in a new tab). Make sure to install it to the workspace you want to send notifications to.

Once you've created and installed the application, you'll need to get the bot token that will be used to send notifications. You can find it under the OAuth & Permissions tab in the application settings.

It will look something like this: xoxb-*****.

Create a configuration file for Alertmanager

Create a file called alertmanager.yml with the contents below.

  resolve_timeout: 1m
  receiver: "slack-notifications"
  - name: "slack-notifications"
      - channel: "#alerts" # replace with whichever Slack channel is used
        send_resolved: true
            credentials: xoxb-my-bot-token

The Alertmanager works by defining a set of receivers that will be triggered when an alert is triggered. In this case, we're defining a single receiver called slack-notifications that will send a notification to Slack.

Alertmanager configuration can be updated at runtime without restarting the service: you can simply update the configuration file alertmanager.yml and send a POST request to the <alertmanager_host>/-/reload endpoint to reload the configuration.

You can read more about the configuration options available in the Alertmanager documentation (opens in a new tab).

Run the Docker image

Run the following command to start the Alertmanager Docker image:

docker run \
    -p 9093:9093 \
    -v $(pwd)/alertmanager.yml:/etc/alertmanager/alertmanager.yml \

This will start the Alertmanager service on port 9093 and mount the configuration file we created earlier.

Update Prometheus configuration

Now that we have the Alertmanager running, we need to update the Prometheus configuration to point to the Alertmanager instance we just started.

Open the Prometheus configuration file prometheus.yml and add the following configuration:

  # ...
    - static_configs:
        - targets:
            - "alertmanager:9093" # or the host address of the Alertmanager instance

Add the Autometrics Slack App

Autometrics Slack App is a simple open-source, self-hosted Slack service for Autometrics SLO alerts. It integrates with the Alertmanager and adds more context for each triggered message: the Slack app pulls recent relevant Prometheus data and renders it as an image of the graph on Slack.

Autometrics Slack app is open-source and available as a Docker image (opens in a new tab). Here's a quick guide how to add it and configure it with your Prometheus and Alertmanager setup.

Create a new Slack application

Go to (opens in a new tab), select your workspace, and create an application from manifest.

This is a basic Slack application manifest with limited permissions to post chat messages on channels that the application gets added to. You can find the manifest below:

  major_version: 1
  minor_version: 0
  name: Autometrics
  description: This bot provides a way for our Autometrics slack-app to communicate with the Slack API.
  long_description: |
    This bot provides a way for our Autometrics slack-app to communicate with
    the Slack API. The autometrics slack-app will post messages to a specified
    channel whenever a alert occurs in your alertmanager (note this requires
    modifications in your alertmanager configuration).
  background_color: "#00FFAA"
  org_deploy_enabled: false
  socket_mode_enabled: false
  token_rotation_enabled: false
    home_tab_enabled: false
    messages_tab_enabled: false
    display_name: Autometrics
    always_online: false
      - chat:write

Install your new Slack application into your workspace in the Install your app section and grab the Bot User OAuth Token, starts with xoxb-***

Deploy the Slack application

You can deploy the Slack application as any other Docker image. The Slack application requires some configuration passed in as environment variables and needs access to the Prometheus, Alertmanager and Slack API.

Here are the necessary environment variables:

LISTEN_HOSTMake sure this address allows for remote connections
BASE_URLThe URL that the Slack should be accessible at (for Slack API)
SLACK_CHANNELWhich Slack channel the app should post alerts to (make sure the application is invited to it!)
SLACK_BOT_TOKENThe xoxb-*** value we grabbed earlier when creating the application
STORAGE_DIRThe directory where the generated alert images should be stored. Make sure it's persistent storage
DB_CONNECTION_STRINGThe sqlite connection string for historic alert storage. Make sure it's persistent

Here's an example docker-compose.yaml file with the above configuration in place:

version: '3.8'
    image: autometrics/slack-app:v0.1.0
      - "3031:3031"
      RUST_LOG: info
      LOG_JSON: "true"
      PORT: "3031"
      BASE_URL: ""
      SLACK_CHANNEL: "test-slack-app"
      SLACK_BOT_TOKEN: xoxb-000000000-000000000-000000000
      PROMETHEUS_URL: http://prometheus:9090
      STORAGE_DIR: /data/
      DB_CONNECTION_STRING: "sqlite:///data/slack-app.db?mode=rwc"
      - slack-app-data:/data

Update Alertmanager configuration

Finally we need to update the Alertmanager so that it sends a Webhook request whenever an alert is triggered to our newly deployed Slack application:

- name: default-receiver
  - send_resolved: true
    url: 'http://slack-app:3031/api/alerts'
  receiver: default-receiver