Deploying Prometheus in Kubernetes

Prometheus is the most popular open-source monitoring solution for Kubernetes, therefore deploying and configuring it is pretty straightforward.

Prometheus supports service discovery, so it can automatically find Kubernetes pods to scrape.

Requirements

Kubernetes cluster running with kubectl configured to access it
Make sure you have Helm (opens in a new tab) installed

Instructions

Create a `monitoring` namespace

It is a good practice to keep all monitoring resources in a separate namespace.

kubectl create namespace monitoring

Deploy Prometheus to Kubernetes

Run the following command to install Prometheus using the Helm chart:

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

This will install Prometheus in your cluster along with Alertmanager, Node Exporter, and a few other components.

Make Autometrics discoverable

If you have installed Prometheus using the helm chart (opens in a new tab), service discovery should be enabled by default.

Service discovery uses Kubernetes annotations to tell Prometheus which ports to look for. In order to make sure that your application metrics are available add the following annotations to your app's deployment:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9464" # this can be different depending on which library you have used for instrumentation

(Optional) Add Autometrics ruleset to Prometheus

If you want to make use of the SLO alerting features in Autometrics, you need to add the Autometrics alerting rules to your Prometheus instance.

You can find the ruleset in the Autometrics repository (opens in a new tab).

Save the ruleset to a file

curl https://raw.githubusercontent.com/autometrics-dev/autometrics-shared/main/autometrics.rules.yml > autometrics.rules.yml

Create a ConfigMap from the ruleset

kubectl create configmap autometrics-rules --from-file=autometrics.rules.yml --namespace monitoring

Create a `values.yaml` file for the Prometheus Helm chart

The values.yaml file is used to add extra configuration to the Prometheus Helm chart. In this case we will mount the extra configuration as a ConfigMap and modify Prometheus configuration to include the Autometrics ruleset.

values.yaml

server:
  extraConfigmapMounts:
    - name: autometrics-rules
      mountPath: /autometrics-alerts/
      configMap: autometrics-rules
      readOnly: true
serverFiles:
  prometheus.yml:
    rule_files:
      - /etc/config/recording_rules.yml # leave whichever rules you have already
      - /etc/config/alerting_rules.yml #
      - /autometrics-alerts/autometrics.rules.yml # <- add the autometrics ruleset here

Upgrade Prometheus using Helm

Upgrade the Prometheus installation using the helm upgrade command, passing in the values.yaml file.

helm upgrade prometheus prometheus-community/kube-prometheus-stack --namespace monitoring -f values.yaml

Docker Northflank