Labels and annotations
Labels and annotations add additional information about an alert using key/value pairs:
- Labels are used to differentiate an alert from all other alerts.
- Annotations are used to provide extra detail on an existing alert.
Labels
Labels are unique identifiers of an alert instance. You can use them for searching, silencing, and routing notifications.
Examples of labels are server=server1 or team=backend. Each alert rule can have more than one label and the complete set of labels for an alert rule is called its label set. It is this label set that identifies the alert.
For example, an alert instance might have the label set {alertname="High CPU usage",server="server1"} while another alert instance might have the label set {alertname="High CPU usage",server="server2"}. These are two separate alert instances because although their alertname labels are the same, their server labels are different.
Labels are a fundamental component of alerting:
- The complete set of labels for an alert is what uniquely identifies an alert instance.
- The alerting UI shows labels for every alert instance generated during evaluation of that rule.
- Notification policies and silences use labels to match alert instances and route them to contact points or stop their notifications.
- Contact points can include information from labels in notification messages.
Label types
An alert’s label set can contain three types of labels:
User-configured labels
Labels that you manually configure in the alert rule to identify the generated alert instances or group them.
You can also use a template to customize the label value and generate dynamic values when the rule is evaluated.
Data source query labels
For example, if you are monitoring temperature readings and each time series for these readings has a sensor_id, and a location label, an alert instance might have the labels {sensor_id="1",location="NY"}, while another alert instance might have {sensor_id="2",location="WA"}.
Data source query labels labels are also used to generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data.
Reserved labels
Reserved labels are automatically added by Grafana:
alert_name: the name of the alert rule.grafana_folder: the title of the folder containing the alert.
Labels prefixed with grafana_ are reserved by Grafana for special use. To stop Grafana Alerting from adding a reserved label, you can disable it via the disabled_labels option in unified_alerting.reserved_labels configuration.
Note
Two alert rules cannot produce alert instances with the same labels. If two alert rules have the same labels such as
foo=bar,bar=bazandfoo=bar,bar=bazthen one of the generated alert instances will be discarded.Ensure the label set for an alert does not have two or more labels with the same name.
- If a configured label has the same name as a data source query label, it replaces the data source label.
- If a configured label has the same name as a reserved label, it is omitted.
Grafana’s built-in Alertmanager supports both Unicode label keys and values. If you are using an external Prometheus Alertmanager, label keys must be compatible with their data model.
This means that label keys must only contain ASCII letters, numbers, as well as underscores and match the regex [a-zA-Z_][a-zA-Z0-9_]*.
Any invalid characters will be removed or replaced by the Grafana alerting engine before being sent to the external Alertmanager according to the following rules:
Whitespacewill be removed.ASCII characterswill be replaced with_.All other characterswill be replaced with their lower-case hex representation. If this is the first character it will be prefixed with_.
Example: A label key/value pair Alert! 🔔="🔥" will become Alert_0x1f514="🔥".
If multiple label keys are sanitized to the same value, the duplicates will have a short hash of the original label appended as a suffix.
Annotations
The purpose of annotations is to add additional information to alert instances, such as extra details for notification messages.
Grafana provides several optional annotations that you can edit for use in notification messages and within Grafana:
summary: A short summary of what the alert has detected and why.description: A detailed description of what happened and what the alert does.runbook_url: The runbook page to guide operators managing a potential incident.dashboardUIdandpanelId: Link the alert to a dashboard and panel. These are automatically set when creating an alert from panels.
Like labels, annotations can use a template to customize the label value and generate dynamic values when the rule is evaluated.
Was this page helpful?
Related resources from Grafana Labs
