DocumentationGrafana MimirConfigureOpenTelemetry Collector

Configure the OpenTelemetry Collector to write metrics into Mimir

When using the OpenTelemetry Collector, you can write metrics into Mimir via two options: prometheusremotewrite and otlphttp.

We recommend using the prometheusremotewrite exporter when possible because the remote write ingest path is tested and proven at scale.

Remote Write

For the Remote Write, use the prometheusremotewrite exporter in the Collector:

In the exporters section add:

yaml Copy

exporters:
  prometheusremotewrite:
    endpoint: http://<mimir-endpoint>/api/v1/push

And enable it in the service.pipelines:

yaml Copy

service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., prometheusremotewrite]

If you want to authenticate using basic auth, we recommend the basicauth extension:

yaml Copy

extensions:
  basicauth/prw:
    client_auth:
      username: username
      password: password

exporters:
  prometheusremotewrite:
    auth:
      authenticator: basicauth/prw
    endpoint: http://<mimir-endpoint>/api/v1/push

service:
  extensions: [basicauth/prw]
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., prometheusremotewrite]

OTLP

Mimir supports native OTLP over HTTP. To configure the collector to use the OTLP interface, you use the otlphttp exporter:

yaml Copy

exporters:
  otlphttp:
    endpoint: http://<mimir-endpoint>/otlp

And enable it in service.pipelines:

yaml Copy

service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

If you want to authenticate using basic auth, we recommend the basicauth extension:

yaml Copy

extensions:
  basicauth/otlp:
    client_auth:
      username: username
      password: password

exporters:
  otlphttp:
    auth:
      authenticator: basicauth/otlp
    endpoint: http://<mimir-endpoint>/otlp

service:
  extensions: [basicauth/otlp]
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

Format considerations

We follow the official OTLP Metric points to Prometheus specification.

By default, Grafana Mimir does not accept OpenTelemetry Exponential Histogram metrics. For Grafana Mimir to accept them, ingestion of Prometheus Native Histogram metrics must first be enabled following the instructions in Configure native histogram ingestion. After this is done, Grafana Mimir will accept OpenTelemetry Exponential Histograms, and convert them into Prometheus Native Histograms following the conventions described in the Exponential Histograms specification.

You might experience the following common issues:

• Dots (.) are converted to _

  Prometheus metrics do not support . and - characters in metric or label names. Prometheus converts these characters to _.

  For example:

  requests.duration{http.status_code=500, cloud.region=us-central1} in OTLP

  requests_duration{http_status_code=”500”, cloud_region=”us-central1”} in Prometheus

• Resource attributes are added to the target_info metric.

  However, <service.namespace>/<service.name> or <service.name> (if the namespace is empty), is added as the label job, and service.instance.id is added as the label instance to every metric.

  For details, see the OpenTelemetry Resource Attributes specification.