Announcing Grafana Mimir, the most scalable open source TSDB in the world
Today we’re introducing you to Grafana Mimir, the most scalable, most performant open source time series database in the world. Mimir allows you to scale to 1 billion metrics and beyond, with simplified deployment, high availability, multi-tenancy, durable storage, and blazing fast query performance that is up to 40x faster than Cortex. Mimir is hosted at https://github.com/grafana/mimir and licensed under AGPLv3.
There’s supposed to be a video here, but for some reason there isn’t. Either we entered the id wrong (oops!), or Vimeo is down. If it’s the latter, we’d expect they’ll be back up and running soon. In the meantime, check out our blog!
Mimir is a new project in the metrics space, but it stands on the shoulders of giants. To understand why Mimir and why now, we need to rewind and tell the history of Cortex.
Born from Prometheus
While working at Weaveworks in 2016, I started the Cortex project with Julius Volz, the co-founder and a fellow maintainer of Prometheus. The goal of the project was to build a scalable Prometheus-compatible solution, designed to be run as a SaaS offering. After I joined Grafana Labs, we worked with Weaveworks to move Cortex to a neutral ground, the Cloud Native Computing Foundation. Cortex was accepted as a CNCF sandbox project on September 20, 2018, and promoted to an incubating project two years later. The CNCF offers a level playing field for organizations to collaborate on the project, and this really worked well, with both Grafana Labs and Weaveworks actively contributing. Cortex is used by more than 20 organizations and has had contributions from around 100 developers. Grafana Labs employees have been far and away the largest contributors to the Cortex project, with ~87% of the commits from 2019-2021.
Source: cortex.devstats.cncf.io
Cortex, Loki, Tempo, and Grafana Enterprise Metrics
Over time, Cortex has become the basis for many other projects, including Grafana Loki (“like Prometheus, but for logs”), Grafana Tempo (for distributed tracing), and Grafana Enterprise Metrics (GEM). Grafana Labs introduced this product in 2020 to bring Prometheus to large organizations, adding features needed to make it enterprise-ready — things like additional security, access controls, and simplified management UIs — with the aim of selling to organizations that didn’t have the time or the desire to build this themselves. The goal was to make GEM the “easy button” that checked all enterprise requirements from day one.
At the same time, cloud providers and ISVs have introduced their own offerings based on Cortex, without similar levels of contribution to the project. The cycle of one company building technology that others can offer at a lower cost, because they don’t have to invest in the open source code, is simply not sustainable. In response, we started investing more in GEM than in Cortex. We are a company with a bias for open source, so this is something that has made us super uncomfortable; some of the scalability- and performance-related features we built into GEM were features we truly believed should be OSS.
As many know, last year we relicensed our open source projects, Grafana, Grafana Loki, and Grafana Tempo, from Apache 2.0 to AGPLv3, an OSI-approved license that preserves open source freedoms while encouraging third parties to contribute code back to the community. From the beginning of Grafana Labs, our goal was to build a sustainable business around our open source projects, so that revenue from our commercial offerings could be re-invested in the technology and the community. The AGPL license helps balance the “value creation” of open source and the community with the “value capture” of our monetization strategy.
Introducing Grafana Mimir
Mimir combines the best of what we built in Cortex with features we developed to run GEM and Grafana Cloud at massive scale, all under the AGPLv3 license. Included with Mimir are previously commercial features, including unlimited cardinality using a horizontally scalable, “split” compactor and blazing fast, high cardinality queries through a sharded query engine.
Comparing Cortex, Grafana Mimir, and Grafana Cloud & Grafana Enterprise Metrics
In the process of building Mimir and forking Cortex, we had the opportunity to chip away at five years of accumulated technical debt, removing unused features, making the project easier to maintain, dramatically simplifying the configuration, and improving our documentation. Our hope is that this investment will help the community grow by making Mimir massively easier to adopt.
For our Grafana Cloud and Grafana Enterprise Metrics customers, nothing changes — both products have been based on Grafana Mimir for months. For organizations currently running Cortex, Mimir can be a drop-in replacement within the constraints of a major version upgrade. It takes less than 10 minutes to migrate from Cortex to Mimir for most setups.
The future of metrics
The vision for Mimir is not to be “the best scalable Prometheus,” but to be “the best scalable time series database regardless of metrics format.” Users shouldn’t have to change their code to send metrics to Mimir. Today, Mimir can natively consume Prometheus metrics. And soon, Influx, Graphite, OpenTelemetry, and Datadog will follow. This is part of our “big tent” philosophy: Just as Grafana is the one tool to visualize all your data, Mimir can be the one tool to store all your metrics.
Now that we’ve released Mimir, users everywhere can put together the pieces of what I think is the most powerful, most comprehensive, and most composable open source observability stack, the LGTM stack: Loki for logs, Grafana for visualization, Tempo for tracing, and Mimir for metrics.
To learn more, read the Q&A with our CEO, Raj Dutt, and register for the April 26 webinar, Intro to Grafana Mimir, the open source time series database that scales to 1 billion metrics & beyond.
