7 ms·

CNCF's Cortex v1.0: scalable, fast Prometheus implementation

181 points by netingle 6y ago

netingle 6y ago
Hi! Tom, one of the Cortex authors here. Super proud of the team and this release - let me know if you have any questions!
- ctovena 6y ago
  Great job Cortex team, Do you think this means Cortex will move to incubation in the CNCF landscape ?
  - netingle 6y ago
    I hope so! Goutham is apply for incubation as we speak..
    - RichiH 6y ago
      This will also depend on SIG o11y, the creation of which is currently being voted on by CNCF TOC. TOC vote is looking good and projects which have been in sandbox for some time are obvious candidates for early review.
- number101010 6y ago
  Hey Tom! Can you outline how Cortex differs from some of the other available Prometheus backends?
  - netingle 6y ago
    Sure, check out this talk from PromCon I did with Bartek, the Thanos author: https://grafana.com/blog/2019/11/21/promcon-recap-two-households-both-alike-in-dignity-cortex-and-thanos/ https://grafana.com/blog/2019/11/21/promcon-recap-two-househ...
    - MetalMatze 6y ago
      Love that talk. :)
  - valyala 6y ago
    VictoriaMetrics FAQ contains comparisons to Cortex [1] and Thanos [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-cortex https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#... [2] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-thanos https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#...
rfratto 6y ago
Great job Cortex team!
mattmendick 6y ago
Really exciting! Well done
ones_and_zeros 6y ago
Isn't prometheus an implementation and not an interface? I have "prometheus" running in my cluster, if it's not cortex, what implementation am I using?
- gouthamve 6y ago
  Yes, you're running the Prometheus server. But what Cortex is a Prometheus API compatible service that horizontally scales and has multi-tenancy and other things built in.
- netingle 6y ago
  Yes, Prometheus is an implementation - the HN text has a limited number of words, so I thought "Prometheus implementation" conveyed the fact Cortex was trying to be a 100% API compatible implementation of Prometheus, but with scalability, replication etc
  - cat199 6y ago
    how about: CNCF's Cortex v1.0: scalable, fast Prometheus API implementation ready for prod (grafana.com) saves 1 char.
- ownagefool 6y ago
  It's kinda several things - The OSS product - The Storage Format (I guess) - The Interface for pulling metrics (https://github.com/OpenObservability/OpenMetrics https://github.com/OpenObservability/OpenMetrics) I haven't dug into cortex even a little, but the other comments are suggesting it's API compatible but essentially claiming they're production ready because they'll give you things the OSS project won't give you out of the box, i.e. long term storage and RBAC. Looks like a good thing.
  - netingle 6y ago
    > wrapping prometheus and giving you that production readyness that they're claiming the OSS project won't give you out of the box No! Prometheus is and has been production ready for many years. Cortex is a clustered/horizontally scalable implemention of the Prometheus APIs, and Cortex has just gone production ready. Sorry for the confusion.
stuff4ben 6y ago
This was a Weaveworks project right?
- gouthamve 6y ago
  Yes, it was created at Weaveworks, but it was later donated to CNCF and now the community is much bigger! Having said that Weaveworks is still a major contributor!
kapilvt 6y ago
also props to https://weave.works https://weave.works for creating cortex, open-sourcing it and moving it under cncf, something this blog post leaves out.
Rapzid 6y ago
Dat architecture tho: https://cortexmetrics.io/docs/architecture/ https://cortexmetrics.io/docs/architecture/ . Holy bi-gebus.
- netingle 6y ago
  Thats the "microservices" mode - you can run it as a single process and the architecture becomes super boring. Its like looking at the module interdependencies of reasonably large piece of software; of course its going to look complicated.
  - valyala 6y ago
    According to Cortex docs [1], a single-process Cortex isn't production ready. It is intended for development and testing only. [1] https://cortexmetrics.io/docs/configuration/single-process-config/ https://cortexmetrics.io/docs/configuration/single-process-c...
nopzor 6y ago
awesome job by the cortex team! there's a lot of good questions, and some confusion in this thread. here is my view. note: i'm definitely biased; am the co-founder/ceo at grafana labs. - at grafana labs we are huge fans of prometheus. it has become the most popular metrics backend for grafana. we view cortex and prometheus as complementary. we are also very active contributors to the prometheus project itself. in fact, cortex vendors in prometheus. - you can think of cortex as a scale-out, multi-tenant, highly available "implementation" of prometheus itself. - the reason grafana labs put so much resources into cortex is because it powers our grafana cloud product (which offers a prometheus backend). like grafana itself, we are also actively working on an enterprise edition of cortex that is designed to meet the security and feature requirements of the largest companies in the world. - yes, cortex was born at weaveworks in 2016. tom wilkie (vp of product at grafana labs) co-created it while he worked there. after tom joined grafana labs in 2018, we decided to pour a lot more resources into the project, and managed to convince weave.works to move it to the cncf. this was a great move for the project and the community, and cortex has come a long long way in the last 2 years. once again, a big hat tip to everyone who made this release possible. a big day for the project, and for prometheus users in general! [edit: typos]
- Florin_Andrei 6y ago
  I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-started-chunks-storage/ https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?
  - netingle 6y ago
    There are a bunch of different solutions out there; Thanos, Influx, federated Prometheus etc. The local Cortex storage works pretty well but we have a very high bar for production worthiness. Right now I'd recommend using Bigtable of DynamoDB, and if you're on premise Cassandra. In the future the block storage will allow you to run minio.
  - 6y ago
throwaway50203 6y ago
Reminder: github star history is in no way a measure of quality.
zytek 6y ago
Congrats to Grafana Team! If you're looking at scaling your Prometheus setup - check out also Victoria Metrics. Operational simplicity and scalability/robustness are what drive me to it. I used to to send metrics from multiple Kubernetes clusters with Prometheus - each cluster having Prom with remote_write directive to send metrics to central VictoriaMetrics service. That way my "edge" prometheus installations are practically "stateless", easily set up using prometheus-operator. You don't even need to add persistent storage to them.
mmcclellan 6y ago
New to Cortex but when looking at a comparison of Prometheus and InfluxDB (like https://prometheus.io/docs/introduction/comparison/#prometheus-vs-influxdb https://prometheus.io/docs/introduction/comparison/#promethe...) it appears that Cortex offers similar horizontal scalability features to the InfluxDB Enterprise offering. The linked comparison does note the difference between event logging and metrics recording but I am curious (choosy beggar that I am) whether others consider them separate tooling or whether it is possible to remain performant using one solution.
demilich 6y ago
Good job, excited!

181 points by netingle 6y ago

Hi! Tom, one of the Cortex authors here. Super proud of the team and this release - let me know if you have any questions!

ctovena 6y ago
Great job Cortex team, Do you think this means Cortex will move to incubation in the CNCF landscape ?
- netingle 6y ago
  I hope so! Goutham is apply for incubation as we speak..
  - RichiH 6y ago
    This will also depend on SIG o11y, the creation of which is currently being voted on by CNCF TOC. TOC vote is looking good and projects which have been in sandbox for some time are obvious candidates for early review.
number101010 6y ago
Hey Tom! Can you outline how Cortex differs from some of the other available Prometheus backends?
- netingle 6y ago
  Sure, check out this talk from PromCon I did with Bartek, the Thanos author: https://grafana.com/blog/2019/11/21/promcon-recap-two-households-both-alike-in-dignity-cortex-and-thanos/ https://grafana.com/blog/2019/11/21/promcon-recap-two-househ...
  - MetalMatze 6y ago
    Love that talk. :)
- valyala 6y ago
  VictoriaMetrics FAQ contains comparisons to Cortex [1] and Thanos [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-cortex https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#... [2] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-thanos https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#...

rfratto 6y ago

Great job Cortex team!

mattmendick 6y ago

Really exciting! Well done

ones_and_zeros 6y ago

Isn't prometheus an implementation and not an interface? I have "prometheus" running in my cluster, if it's not cortex, what implementation am I using?

gouthamve 6y ago
Yes, you're running the Prometheus server. But what Cortex is a Prometheus API compatible service that horizontally scales and has multi-tenancy and other things built in.
netingle 6y ago
Yes, Prometheus is an implementation - the HN text has a limited number of words, so I thought "Prometheus implementation" conveyed the fact Cortex was trying to be a 100% API compatible implementation of Prometheus, but with scalability, replication etc
- cat199 6y ago
  how about: CNCF's Cortex v1.0: scalable, fast Prometheus API implementation ready for prod (grafana.com) saves 1 char.
ownagefool 6y ago
It's kinda several things - The OSS product - The Storage Format (I guess) - The Interface for pulling metrics (https://github.com/OpenObservability/OpenMetrics https://github.com/OpenObservability/OpenMetrics) I haven't dug into cortex even a little, but the other comments are suggesting it's API compatible but essentially claiming they're production ready because they'll give you things the OSS project won't give you out of the box, i.e. long term storage and RBAC. Looks like a good thing.
- netingle 6y ago
  > wrapping prometheus and giving you that production readyness that they're claiming the OSS project won't give you out of the box No! Prometheus is and has been production ready for many years. Cortex is a clustered/horizontally scalable implemention of the Prometheus APIs, and Cortex has just gone production ready. Sorry for the confusion.

stuff4ben 6y ago

This was a Weaveworks project right?

gouthamve 6y ago
Yes, it was created at Weaveworks, but it was later donated to CNCF and now the community is much bigger! Having said that Weaveworks is still a major contributor!

kapilvt 6y ago

also props to https://weave.works https://weave.works for creating cortex, open-sourcing it and moving it under cncf, something this blog post leaves out.

Rapzid 6y ago

Dat architecture tho: https://cortexmetrics.io/docs/architecture/ https://cortexmetrics.io/docs/architecture/ . Holy bi-gebus.

netingle 6y ago
Thats the "microservices" mode - you can run it as a single process and the architecture becomes super boring. Its like looking at the module interdependencies of reasonably large piece of software; of course its going to look complicated.
- valyala 6y ago
  According to Cortex docs [1], a single-process Cortex isn't production ready. It is intended for development and testing only. [1] https://cortexmetrics.io/docs/configuration/single-process-config/ https://cortexmetrics.io/docs/configuration/single-process-c...

nopzor 6y ago

awesome job by the cortex team! there's a lot of good questions, and some confusion in this thread. here is my view. note: i'm definitely biased; am the co-founder/ceo at grafana labs. - at grafana labs we are huge fans of prometheus. it has become the most popular metrics backend for grafana. we view cortex and prometheus as complementary. we are also very active contributors to the prometheus project itself. in fact, cortex vendors in prometheus. - you can think of cortex as a scale-out, multi-tenant, highly available "implementation" of prometheus itself. - the reason grafana labs put so much resources into cortex is because it powers our grafana cloud product (which offers a prometheus backend). like grafana itself, we are also actively working on an enterprise edition of cortex that is designed to meet the security and feature requirements of the largest companies in the world. - yes, cortex was born at weaveworks in 2016. tom wilkie (vp of product at grafana labs) co-created it while he worked there. after tom joined grafana labs in 2018, we decided to pour a lot more resources into the project, and managed to convince weave.works to move it to the cncf. this was a great move for the project and the community, and cortex has come a long long way in the last 2 years. once again, a big hat tip to everyone who made this release possible. a big day for the project, and for prometheus users in general! [edit: typos]

Florin_Andrei 6y ago
I'm worried about this statement: > Local storage is explicitly not production ready at this time. https://cortexmetrics.io/docs/getting-started/getting-started-chunks-storage/ https://cortexmetrics.io/docs/getting-started/getting-starte... But I want a scale-out, multitenant implementation of Prometheus with local storage that's ready for prod. What are my options then? VictoriaMetrics?
- netingle 6y ago
  There are a bunch of different solutions out there; Thanos, Influx, federated Prometheus etc. The local Cortex storage works pretty well but we have a very high bar for production worthiness. Right now I'd recommend using Bigtable of DynamoDB, and if you're on premise Cassandra. In the future the block storage will allow you to run minio.
- 6y ago

throwaway50203 6y ago

Reminder: github star history is in no way a measure of quality.

zytek 6y ago

Congrats to Grafana Team! If you're looking at scaling your Prometheus setup - check out also Victoria Metrics. Operational simplicity and scalability/robustness are what drive me to it. I used to to send metrics from multiple Kubernetes clusters with Prometheus - each cluster having Prom with remote_write directive to send metrics to central VictoriaMetrics service. That way my "edge" prometheus installations are practically "stateless", easily set up using prometheus-operator. You don't even need to add persistent storage to them.

mmcclellan 6y ago

New to Cortex but when looking at a comparison of Prometheus and InfluxDB (like https://prometheus.io/docs/introduction/comparison/#prometheus-vs-influxdb https://prometheus.io/docs/introduction/comparison/#promethe...) it appears that Cortex offers similar horizontal scalability features to the InfluxDB Enterprise offering. The linked comparison does note the difference between event logging and metrics recording but I am curious (choosy beggar that I am) whether others consider them separate tooling or whether it is possible to remain performant using one solution.

demilich 6y ago

Good job, excited!