DocumentationGrafana LokiRelease notesv3.1

v3.1

Grafana Labs and the Loki team are excited to announce the release of Loki 3.1. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

Key features in Loki 3.1.0 include the following:

• Query acceleration with Bloom filters (experimental): Continued work on improving the new Bloom filters feature introduced in Loki 3.0. This is designed to speed up filter queries, with best results for queries that are looking for a specific text string like an error message or UUID. For more information, refer to Query acceleration with Blooms.

• Helm: Update Helm chart to support distributed mode and 3.0 (#12067) (79b876b). Add pattern ingester support in SSD mode (#12685) (19bfef4). Enable log volume endpoint by default in Helm (#12690) (e39677f). Allow extraObject items as multiline strings (#12397) (af5be90).Support for PVC Annotations for Non-Distributed Modes (#12023) (efdae3d).

• LogQL: Support negative numbers in LogQL (#13091) (6df81db). Improve performance of first_over_time and last_over_time queries by sharding them (#11605) (f66172e). Improve syntax parser for pattern (#12489) (48dae44).

• lokitool: Add lokitool to replace cortextool. (#12166) (7b7d3d4). Introduce index audit to lokitool (#13008) (47f0236).

• Explore Logs: Explore Logs, which lets you explore your Loki data without writing LogQL queries, is now available in public preview. If you are a Grafana Cloud user, you can access Explore Logs in the Grafana Cloud main navigation menu. If you are not a Grafana Cloud user, you can install the Explore Logs plugin. For more information, refer to the Explore Logs documentation.

• Docs: Added a video to the Getting Started demo and updated for Grafana Alloy. Added an interactive sandbox to the Loki Quickstart tutorial. Updated the documentation for the SSD and microservices deployment modes using the Helm charts. Documented the new meta-monitoring Helm chart.

Other improvements include the following:

• API: Add step param to Patterns Query API (#12703) (7b8533e).
• API: Expose optional label matcher for label names API (#11982) (8084259).
• cache: Add Cache-Control: no-cache support for Loki instant queries. (#12896) (88e545f).
• canary: Add test to check query results with and without cache. (#13104) (71507a2).
• distributor: Optimize log parsing performance by using unsafe package (#13223) (9f31b25).
• Enable log volume endpoint by default (#12628) (397aa56).
• ingester: Add backoff to flush op (#13140) (9767807).
• ingester: Add ingester_chunks_flush_failures_total (#12925) (cc3694e).
• ingester: add profile tagging to ingester (#13068) (00d3c7a).
• ingester: add recalculateOwnedStreams to check stream ownership if the ring is changed (#13103) (e7689b2).
• ingester: Add profile tagging to ingester (#13068) (00d3c7a).
• ingester: Include the stream we failed to create in the stream limit error message (#12437) (ec81991).
• ingester: New stream count limiter (#13006) (1111595).
• Lambda-Promtail: Parameterise the MaximumEventAgeInSeconds, LogGroupName, and IAMRoleName for Lambda-Promtail CloudFormation template (#12728) (8892dc8).
• loki/main.go: Log which config file path is used on startup (#12985) (7a3338e).
• metamonitoring: Add a version of the mixin dashboards for meta monitoring (#12700) (ec1a057).
• Promtail: Added support to install wget on Promtail docker image to support docker healthcheck (#11711) (ffe684c).
• Promtail: Support of RFC3164 aka BSD Syslog (#12810) (be41525).
• querier: Split gRPC client into two. (#12726) (7b6f057).
• reporting: Report CPU usage (#12970) (87288d3).
• Split detected fields queries (#12491) (6c33809).
• TSDB: Add fast-path to inversePostingsForMatcher (#12679) (402d1d7).

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Deprecated storage options including the deprecation of the BoltDB store.

• Deprecated configuration options.

• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

Upgrade Considerations

he path from 2.9 to 3.0 included several breaking changes. For important upgrade guidance, refer to the Upgrade Guide and the separate Helm Upgrade Guide.

• BREAKING CHANGE Update Helm chart to support distributed mode and 3.0 (#12067).

Bug fixes

3.1.0 (2024-07-02)

• blooms: Fix duplicate enqueue item problem in bloom download queue when do sync download (#13114) (f98ff7f).
• blooms: Add a missing continue in fuse which may cause incorrect bloom test result (#12650) (0d1ebeb).
• blooms: Allow cluster label override in bloom dashboards (#13012) (987e551).
• blooms: bloomshipper no longer returns empty metas on fetch (#13130) (ad279e5).
• blooms: Clean block directories recursively on startup (#12895) (7b77e31).
• blooms: Correctly return unfiltered chunks for series that are not mapped to any block (#12774) (c36b114).
• blooms: Deduplicate filtered series and chunks (#12791) (3bf2d1f).
• blooms: Disable metas cache on bloom gateway (#12959) (00bdd2f).
• blooms: Do not fail requests when fetching metas from cache fails (#12838) (667076d).
• blooms: Don’t break iterator conventions (#12808) (1665e85).
• blooms: Fix partitionSeriesByDay function (#12900) (738c274).
• blooms: Fix a regression introduced with #12774 (#12776) (ecefb49).
• blooms: Fix findGaps when ownership goes to MaxUInt64 and that is covered by existing meta (#12558) (0ee2a61).
• blooms: Fully deduplicate chunks from FilterChunkRef responses (#12807) (a0f358f).
• blooms: Handle not found metas gracefully (#12853) (37c8822).
• blooms: Reset error on LazyBloomIter.Seek (#12806) (76ba24e).
• blooms: Defer closing blocks iter after checking error from loadWorkForGap (#12934) (cb1f5d9).
• blooms: Fix bloom deleter PR after merge (#13167) (c996349).
• blooms: Fix incorrect sorting of chunks in bloom-filtered response since ChunkRef.Cmp method is used in reverse (#12999) (670cd89).
• blooms: Remove unused parameter causing lint error (#12801) (33e82ec).
• codec not initialized in downstream roundtripper (#12873) (b6049f6).
• Change log level since this is a known case (#13029) (ca030a5).
• Close res body (#12444) (616977a).
• Correct initialization of a few slices (#12674) (0eba448).
• distributor: add detected_level info when the info word appears on log message (#13218) (c9bfa3e).
• docs: Broken link in getting started readme (#12736) (425a2d6).
• docs: Move Promtail configuration to the correct doc (#12737) (1161846).
• docs: Updated Loki OTLP Ingest Configuration (#12648) (ff88f3c).
• Explore logs: Make detected fields work for both json and proto (#12682) (f68d1f7).
• Fix errors reported by the race detector (#13174) (2b19dac), closes #8586.
• Fix for how the loop sync is done (#12941) (5cd850e).
• Fix log level detection (#12651) (6904a65).
• Fix setting of info log level when trying to detect level from log lines (#12635) (0831802).
• lokitool: Fix the lokitool imports (#12673) (6dce988).
• Helm: Fix indentation of query range values in Helm (#12577) (9823f20).
• Helm: Add missing Helm helper loki.hpa.apiVersion (#12755) (3070ea7).
• Helm: Add missing OTLP endpoint to nginx config (#12709) (8096748).
• Helm: Fixes read & backend replicas settings (#12828) (d751134).
• Helm: Set compactor address for distributed mode. (#12748) (521d40a).
• Helm: Fix GEL image tag, bucket name and proxy URLs (#12878) (67ed2f7).
• Helm: Fix query-frontend and ruler targetPort ‘http-metrics’ in Service template (#13024) (1ab9d27).
• Helm: Fix queryScheduler servicemonitor (#12753) (8101e21).
• Helm: Fixed ingress paths mapping (#12932) (5ada92b).
• Helm: Only default bucket names when using MinIO (#12548) (2e32ec5).
• Helm: Removed duplicate bucketNames from documentation and fixed key name deploymentMode (#12641) (0d8ff9e).
• Helm: Ingester zoneAwareReplication (#12659) (9edb0ce).
• Helm: Missing password for Loki-Canary when loki.auth_enabled is true (#12411) (68b23dc).
• indexstats: Do not collect stats from “IndexStats” lookups for other query types (#12978) (1f5291a).
• ingester: Replace channel check with atomic bool in tailer.send() (#12976) (4a5edf1).
• ingester: Not owned stream count (#13030) (4901a5c).
• ingester: Properly return http status codes from ingester to querier for RPC function calls (#13134) (691b174).
• ingester: Track bytes discarded by ingester. (#12981) (88c6711).
• Introduce feature flag for [last|first]_over_time sharding. (#13067) (6e45550).
• Invalidate caches when pipeline wrappers are disabled (#12903) (a772ed7).
• ksonnet: Do not generate RBAC for consul if you are using memberlist (#12688) (2d62fca).
• Lambda-Promtail: Update s3 filename regex to allow finding of log files from AWS GovCloud regions (#12482) (7a81d26).
• Lambda-Promtail: Remove Hardcoded Bucket Name from EventBridge Example CloudFormation Template (#12609) (8c18463).
• LogQL: Fix multiple or filters would get wrong filtertype (#13169) (9981e9e).
• Loki version prefix in Makefile (#12514) (dff72d2).
• Make the TSDB filenames correctly reproducible from the identifier (#12536) (ec888ec).
• metamonitoring: Add missing parentheses in meta monitoring dashboards (#12802) (151d0a5).
• mixins: loki-operational.libsonnet (#12789) (51a841f).
• mixins: Incorrect compactor matcher in loki-deletion dashboard mixin (#12567) (006f88c).
• mixins: Fix compactor matcher in the loki-deletion dashboard (#12790) (a03846b).
• mixin: Mixin generation when cluster label is changed (#12613) (1ba7a30).
• mixin: dashboards $__auto fix (#12707) (91ef72f).
• mixins: Add missing log data source on loki-deletion (#13011) (1948899).
• mixins: Align loki-writes mixins with loki-reads (#13022) (757b776).
• mixins: Remove unnecessary disk panels for SSD read path (#13014) (8d9fb68).
• mixins: Upgrade old plugin for the loki-operational dashboard. (#13016) (d3c9cec).
• nix: Lambda-Promtail vendor hash (#12763) (ae180d6).
• OTLP: Fix parsing of default per tenant OTLP config (#12836) (7cc9a93).
• operator: Add alertmanager client config to ruler template (#13182) (6148c37).
• operator: Bump golang builder to 1.21.9 (#12503) (f680ee0).
• operator: Configure Loki to use virtual-host-style URLs for S3 AWS endpoints (#12469) (0084262).
• operator: Improve API documentation for schema version (#13122) (3a9f50f).
• operator: Use a minimum value for replay memory ceiling (#13066) (4f3ed77).
• Optimize regular initialization (#12926) (a46d14f).
• OTel: Map 500 errors to 503 (#13173) (b31e04e).
• packaging: Require online network in systemd unit file for Loki and Promtail (#12741) (57f78b5).
• Fix panics when ingester response is nil (#12946) (3cc28aa).
• Promtail: Promtail race fixes (#12656) (4e04d07).
• Promtail: Clean up metrics generated from logs after a config reload. (#11882) (39a7181).
• Promtail: Fix bug with Promtail config reloading getting stuck indefinitely (#12795) (4d761ac).
• Promtail: Fix UDP receiver on syslog transport (#10708) (a00f1f1).
• Promtail: Handle docker logs when a log is split in multiple frames (#12374) (c0113db). This changes the current Promtail behavior. Loki has a default max_line_size of 256kb, but With this fix, if the max_line_size config in Promtail has a value, the docker service discovery will now respect that configuration, otherwise it will use the Loki default of 256kb.
• querier: Add retry middleware to the “limited” query roundtripper (#13161) (bb864b3).
• query sharding: Generalize avg -> sum/count sharding using existing binop mapper (#12599) (11e7687).
• regression: reverts grafana/loki #13039 to prevent use-after-free corruptions (#13162) (41c5ee2).
• spans: Corrects early-close for a few spans (#12887) (93aaf29).
• stores: Do not filter out chunks for store when From==Through and From==start (#13117) (d9cc513).
• structured metadata: Add copyString function to symbolizer to avoid retaining memory (#13146) (86b119a).
• Temporarily moving from alloy to alloy dev (#13062) (7ffe0fb).
• Update to build image 0.33.2, fixes bug with Promtail windows DNS resolution (#12732) (759f42d).
• Updated all dockerfiles go1.22 (#12708) (71a8f2c).
• Use to the proper config names in warning messages (#12114) (4a05964).
• workflows: Use an intermediate env variable in GitHub workflow (#12905) (772616c).
• workflows: Don’t run metric collector on forks (#12687) (7253444).