Deploying Alertmanager on Docker

Alertmanager is a utility service deployed alongside Prometheus to help route, group, and prioritize alerts. Alertmanager does not trigger alerts - alerting rules are set in Prometheus itself, Alertmanager only takes care of the lifecycle of the alert after it has been triggered (you can think of it as Zapier but for Prometheus alerts).

Alertmanager is available as a Docker image from the official Prometheus organization.

Requirements

Prometheus

Make sure you have Prometheus running in the environment you're hoping to connect the Alertmanager to. See Deploying Prometheus to Docker.

Autometrics-instrumented application with SLOs

Autometrics enables you to set up smarter alerts using the Service Level Objectives (SLO) approach. You can read more about SLOs in Autometrics here and follow the guides for SLO-instrumentation under each supported language:

Instructions

In the following guide we'll deploy a basic Alertmanager instance configured to work best with Autometrics alerts that will post a notification to Slack whenever there's a breach in the SLO we defined earlier with Autometrics.

Create a Slack application

First, we need to create a Slack application that will be used to send notifications to Slack. You can create a new Slack application here (opens in a new tab). Make sure to install it to the workspace you want to send notifications to.

Once you've created and installed the application, you'll need to get the bot token that will be used to send notifications. You can find it under the OAuth & Permissions tab in the application settings.

It will look something like this: xoxb-*****.

Create a configuration file for Alertmanager

Create a file called alertmanager.yml with the contents below.

alertmanager.yml

global:
  resolve_timeout: 1m
 
route:
  receiver: "slack-notifications"
 
receivers:
  - name: "slack-notifications"
    slack_configs:
      - channel: "#alerts" # replace with whichever Slack channel is used
        send_resolved: true
        api_url: https://slack.com/api/chat.postMessage
        http_config:
          authorization:
            credentials: xoxb-my-bot-token

The Alertmanager works by defining a set of receivers that will be triggered when an alert is triggered. In this case, we're defining a single receiver called slack-notifications that will send a notification to Slack.

Alertmanager configuration can be updated at runtime without restarting the service: you can simply update the configuration file alertmanager.yml and send a POST request to the <alertmanager_host>/-/reload endpoint to reload the configuration.

You can read more about the configuration options available in the Alertmanager documentation (opens in a new tab).

Run the Docker image

Run the following command to start the Alertmanager Docker image:

docker run \
    -p 9093:9093 \
    -v $(pwd)/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
    prom/alertmanager:latest

This will start the Alertmanager service on port 9093 and mount the configuration file we created earlier.

Update Prometheus configuration

Now that we have the Alertmanager running, we need to update the Prometheus configuration to point to the Alertmanager instance we just started.

Open the Prometheus configuration file prometheus.yml and add the following configuration:

prometheus.yml

scrape_configs:
  # ...
 
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager:9093" # or the host address of the Alertmanager instance

Add the Autometrics Slack App

Autometrics Slack App is a simple open-source, self-hosted Slack service for Autometrics SLO alerts. It integrates with the Alertmanager and adds more context for each triggered message: the Slack app pulls recent relevant Prometheus data and renders it as an image of the graph on Slack.

Autometrics Slack app is open-source and available as a Docker image (opens in a new tab). Here's a quick guide how to add it and configure it with your Prometheus and Alertmanager setup.

Create a new Slack application

Go to https://api.slack.com/apps/ (opens in a new tab), select your workspace, and create an application from manifest.

This is a basic Slack application manifest with limited permissions to post chat messages on channels that the application gets added to. You can find the manifest below:

manifest.yml

_metadata:
  major_version: 1
  minor_version: 0
display_information:
  name: Autometrics
  description: This bot provides a way for our Autometrics slack-app to communicate with the Slack API.
  long_description: |
    This bot provides a way for our Autometrics slack-app to communicate with
    the Slack API. The autometrics slack-app will post messages to a specified
    channel whenever a alert occurs in your alertmanager (note this requires
    modifications in your alertmanager configuration).
  background_color: "#00FFAA"
settings:
  org_deploy_enabled: false
  socket_mode_enabled: false
  token_rotation_enabled: false
features:
  app_home:
    home_tab_enabled: false
    messages_tab_enabled: false
  bot_user:
    display_name: Autometrics
    always_online: false
oauth_config:
  scopes:
    bot:
      - chat:write

Install your new Slack application into your workspace in the Install your app section and grab the Bot User OAuth Token, starts with xoxb-***

Deploy the Slack application

You can deploy the Slack application as any other Docker image. The Slack application requires some configuration passed in as environment variables and needs access to the Prometheus, Alertmanager and Slack API.

Here are the necessary environment variables:

Variable	Description
`LISTEN_HOST`	Make sure this address allows for remote connections
`BASE_URL`	The URL that the Slack should be accessible at (for Slack API)
`SLACK_CHANNEL`	Which Slack channel the app should post alerts to (make sure the application is invited to it!)
`SLACK_BOT_TOKEN`	The `xoxb-***` value we grabbed earlier when creating the application
`STORAGE_DIR`	The directory where the generated alert images should be stored. Make sure it's persistent storage
`DB_CONNECTION_STRING`	The sqlite connection string for historic alert storage. Make sure it's persistent

Here's an example docker-compose.yaml file with the above configuration in place:

docker-compose.yaml

version: '3.8'
 
services:
  slack-app:
    image: autometrics/slack-app:v0.1.0
    ports:
      - "3031:3031"
    environment:
      RUST_LOG: info
      LOG_JSON: "true"
      LISTEN_HOST: 0.0.0.0
      PORT: "3031"
      BASE_URL: "https://slack-app.example.com"
      EXPLORER_URL: https://explorer.autometrics.dev
      SLACK_CHANNEL: "test-slack-app"
      SLACK_BOT_TOKEN: xoxb-000000000-000000000-000000000
      PROMETHEUS_URL: http://prometheus:9090
      STORAGE_DIR: /data/
      DB_CONNECTION_STRING: "sqlite:///data/slack-app.db?mode=rwc"
    volumes:
      - slack-app-data:/data
 
volumes:
  slack-app-data:

Update Alertmanager configuration

Finally we need to update the Alertmanager so that it sends a Webhook request whenever an alert is triggered to our newly deployed Slack application:

alertmanager.yml

global:
receivers:
- name: default-receiver
  webhook_configs:
  - send_resolved: true
    url: 'http://slack-app:3031/api/alerts'
route:
  receiver: default-receiver

Railway Kubernetes