Deploying Alertmanager on Docker
Alertmanager is a utility service deployed alongside Prometheus to help route, group, and prioritize alerts. Alertmanager does not trigger alerts - alerting rules are set in Prometheus itself, Alertmanager only takes care of the lifecycle of the alert after it has been triggered (you can think of it as Zapier but for Prometheus alerts).
Alertmanager is available as a Docker image from the official Prometheus organization.
Make sure you have Prometheus running in the environment you're hoping to connect the Alertmanager to. See Deploying Prometheus to Docker.
Autometrics enables you to set up smarter alerts using the Service Level Objectives (SLO) approach. You can read more about SLOs in Autometrics here and follow the guides for SLO-instrumentation under each supported language:
In the following guide we'll deploy a basic Alertmanager instance configured to work best with Autometrics alerts that will post a notification to Slack whenever there's a breach in the SLO we defined earlier with Autometrics.
First, we need to create a Slack application that will be used to send notifications to Slack. You can create a new Slack application here (opens in a new tab). Make sure to install it to the workspace you want to send notifications to.
Once you've created and installed the application, you'll need to get the bot token that will be used to send notifications. You can find it under the
OAuth & Permissions tab in the application settings.
It will look something like this:
Create a file called
alertmanager.yml with the contents below.
- name: "slack-notifications"
- channel: "#alerts" # replace with whichever Slack channel is used
The Alertmanager works by defining a set of
receivers that will be triggered when an alert is triggered. In this case, we're defining a single receiver called
slack-notifications that will send a notification to Slack.
Alertmanager configuration can be updated at runtime without restarting the service: you can simply update the configuration file
alertmanager.yml and send a POST request to the
<alertmanager_host>/-/reload endpoint to reload the configuration.
You can read more about the configuration options available in the Alertmanager documentation (opens in a new tab).
Run the following command to start the Alertmanager Docker image:
docker run \
-p 9093:9093 \
-v $(pwd)/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
This will start the Alertmanager service on port
9093 and mount the configuration file we created earlier.
Now that we have the Alertmanager running, we need to update the Prometheus configuration to point to the Alertmanager instance we just started.
Open the Prometheus configuration file
prometheus.yml and add the following configuration:
- "alertmanager:9093" # or the host address of the Alertmanager instance
Autometrics Slack App is a simple open-source, self-hosted Slack service for Autometrics SLO alerts. It integrates with the Alertmanager and adds more context for each triggered message: the Slack app pulls recent relevant Prometheus data and renders it as an image of the graph on Slack.
Autometrics Slack app is open-source and available as a Docker image (opens in a new tab). Here's a quick guide how to add it and configure it with your Prometheus and Alertmanager setup.
Go to https://api.slack.com/apps/ (opens in a new tab), select your workspace, and create an application from manifest.
This is a basic Slack application manifest with limited permissions to post chat messages on channels that the application gets added to. You can find the manifest below:
description: This bot provides a way for our Autometrics slack-app to communicate with the Slack API.
This bot provides a way for our Autometrics slack-app to communicate with
the Slack API. The autometrics slack-app will post messages to a specified
channel whenever a alert occurs in your alertmanager (note this requires
modifications in your alertmanager configuration).
Install your new Slack application into your workspace in the Install your app section and grab the Bot User OAuth Token, starts with
You can deploy the Slack application as any other Docker image. The Slack application requires some configuration passed in as environment variables and needs access to the Prometheus, Alertmanager and Slack API.
Here are the necessary environment variables:
|Make sure this address allows for remote connections
|The URL that the Slack should be accessible at (for Slack API)
|Which Slack channel the app should post alerts to (make sure the application is invited to it!)
xoxb-*** value we grabbed earlier when creating the application
|The directory where the generated alert images should be stored. Make sure it's persistent storage
|The sqlite connection string for historic alert storage. Make sure it's persistent
Here's an example
docker-compose.yaml file with the above configuration in place:
Finally we need to update the Alertmanager so that it sends a Webhook request whenever an alert is triggered to our newly deployed Slack application:
- name: default-receiver
- send_resolved: true