Skip to content

Create alert with Prometheus

This guide shows you how to create alerts for your application.

0. Prerequisites

You can define alerts by using Kubernetes resources (PrometheusRule), as well as directly in Grafana (GUI based).

You will have a separate alertmanager for each environment available at https://alertmanager.<MY-ENV>.nav.cloud.nais.io/

1. Create PrometheusRule

We use native Prometheus alert rules, and let Alertmanager handle the notifications.

You can define alerts by creating a PrometheusRule resource in your teams namespace.

.nais/alert.yaml 
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: <MY-ALERT>
  namespace: <MY-TEAM>
  labels:
    team: <MY-TEAM>
spec:
  groups:
  - name: <MY-ALERT>
    rules:
    - alert: InstanceDown
      expr: kube_deployment_status_replicas_available{namespace="<namespace>", deployment="<application name>"} == 0
      for: 5m
      annotations:
        consequence: Application is unavailable
        action: "`kubectl describe pod <podname>` -> `kubectl logs <podname>`"
        summary: |-
          This is a multi-line summary with
          linebreaks and everything. Here you can give a more detailed
          summary of what this alert is about
      labels:
        namespace: <MY-TEAM> # required
        severity: critical

2. Activate the alert

Add the file to your application repository, alongside nais.yaml to deploy with NAIS github action.

kubectl apply -f ./nais/alert.yaml

3. Verify your alert

You can see the alerts in the Alertmanager at https://alertmanager.<MY-ENV>.nav.cloud.nais.io/ and the defined rules in Prometheus at https://prometheus.<MY-ENV>.nav.cloud.nais.io/rules

4. Disable resolved (Optional)

A message will be automatically sent when the alert is resolved. In some cases this message may be unnecessary and can be disabled by adding the label send_resolved: "false":

labels:
  namespace: <MY-TEAM> # required
  severity: info
  send_resolved: "false"

Learn how to write good alerts here