7 ms·
Grafana releases OnCall open source project
- Deritio 4y agoI like what grafana labs does with grafana. Im annoyed by their license choice. But apparently when you are grafana everything looks like a dashboard UI? Joke aside I will have a look but I didn't like the screenshots before already. I like the dashboardy thing for dashboards but otherwise it's not a really good UI system for everything else.
- ildari 4y agoHey HN, Ildar here, one of the co-founders of Amixr and one of the software engineers behind Grafana OnCall. Finally we open-sourced the product I'm really excited about that. Please try it out and leave your feedback
- martypitt 4y agoCongrats - this looks great, and definitely something I was wishing for during an incident earlier this week. A minor note, if anyone from Grafana is around - a bunch of the links on the bottom of the announcement go to a 404.
- motakuk 4y agoWe're fixing that, thank you ;)
- anyfactor 4y agoHere is the repo: https://github.com/grafana/oncall https://github.com/grafana/oncall AGPL 3.0
- pphysch 4y agoSeems like a solid replacement for Alertmanager for those already using Grafana OSS. Anyone planning on using both OnCall and Alertmanager?
- dString 4y agoDoesn't AlertManager evaluate metrics and fire alerts? A quick look at OnCall suggests it is more for managing fired alerts than firing alerts. Their own screenshot has AlertManager as an alert source.
- pphysch 4y agoAlertManager does not do the evaluations, it does not connect to any metrics database; those are done by Prometheus/etc and forwarded to AlertManager, which handles deduplication and routing among other things.
- remram 4y agoGrafana used to be so simple, I don't know if I'm a fan of this direction towards many services. Having to run alertmanager and configure it in addition to Grafana was bad enough, now you need to run and configure another service if you want some extra functionalities for those alerts? Are they going to keep maintaining acknowledgements and scheduled silence in AlertManager now that OnCall exists? Are we going to have "legacy notifications" in AlertManager when not running OnCall, the same way there are "legacy alerts" in Grafana when updating from Grafana 7 (pre-AlertManager)?
- juliennakache 4y agoLooking forward to trying this out. I've always felt that PagerDuty was absurdly expensive for the feature set they were offering. It costs something at least $250 per user for organization larger than 5 person - even if you're not an engineer who is ever directly on call. At my previous company, IT had to regularly send surveys to employees to assess if they really needed to have a PagerDuty account. Alerts are a key information in an organization that runs software in production and you shouldn't have to pay $250 / month just to be able to have some visibility into it. I'm hoping Grafana OnCall is able to fully replace PagerDuty.
- CSMastermind 4y ago> I've always felt that PagerDuty was absurdly expensive for the feature set they were offering For anyone out there in the same spot, I'll say that I switched my last company to Atlassian's OpsGenie and it was a 10x cost savings for the same feature set.
- arccy 4y agothe opsgenie api is really bad though if you want to manage it as code/declaratively
- jthrowsitaway 4y agoWe evaluated a bunch of solutions and came to that conclusion as well. Everything we do is in code (Terraform) and will gladly pay for something that has friendly APIs and an already existing Terraform module. Conversely, we'll not engage with or throw away anything that doesn't have friendly APIs.
- dijit 4y agoI really can’t find myself to ever recommend atlassian products though. If cost is the only measure: I understand. But time lost in various areas of the software package (performance alone! Before we get into weird UX paradigms and esoteric query languages, shoddy search systems etc;) surely has an impact on cost. Having your employees spending a lot of time navigating janky software has a cost too.
- this_was_posted 4y agoglad to hear this got open sourced! for someone at grafana; noticed a dead link in the post: https://grafana.com/docs/oncall/main/ https://grafana.com/docs/oncall/main/
- nojito 4y agoUnfortunate that it's AGPL. But this is looks really great!
- ucosty 4y agoWhy is that unfortunate? Unless you're looking to make proprietary changes to Grafana Oncall and host it as a SAAS, it's the same as running any other GPL software.
- nojito 4y agoGPL and its variants are a no go where I work.
- eeZah7Ux 4y agoThen the problem is in the company and not in the license.
- woadwarrior01 4y agoIs Linux verboten at work?
- to11mtm 4y agoProbably not. Linux usually gets a pass, because most times you're just deploying it and not mucking with source code. But a lot of places (I've worked at more that do than don't) will have rules about GPL/AGPL for libraries/infra as a whole though. Often evaluated case-by-case, but it's rare I've seen a AGPL stuff get approved for usage. I think some of it is not wanting to deal with the cost of vigilance; i.e. you can make sure that someone is using %thing% in a way that doesn't run afoul of AGPL right now, but does legal and upper management have confidence in that being true forever and always? Engineers are still human, and corporate management + legal teams tend to hate licensing folk tromping around. This results in refusals ranging from "This is internal for now but we will open it up later" (a fair concern) to "Somebody is worried that exposing it over the VPN to contractors would count as making it public" (IDK, I'm not a lawyer.)
- goodpoint 4y agoIt's very nice to see Python and AGPL used for this.
- ucosty 4y agoLooks very cool, will have to give this a shot.
- motakuk 4y agoHello HN! Matvey Kukuy, ex-CEO of Amixr and a head of the OnCall project here. We've been working hard for a few months to make this OSS release happen. I believe it should make incident response features (on-call rotations, escalations, multi-channel notifications) and best practices more accessible to the wider audience of SRE and DevOps engineers. Hope someone will be able to finally sleep well at night being sure that OnCall will handle escalations and will alert the right person :) Please join our community on a GitHub! The whole Grafana OnCall team is help you and to make this thing better.
- the_duke 4y agoThe docs link [1] is 404. Seems like the /main is the culprit. [1] https://grafana.com/docs/oncall/main/ https://grafana.com/docs/oncall/main/.
- motakuk 4y agoFixed: https://grafana.com/docs/grafana-cloud/oncall/ https://grafana.com/docs/grafana-cloud/oncall/
- knicholes 4y agoBeing on-call has never made me sleep better at night!
- krab 4y agoIf I know someone else is on call and he's competent, I can sleep better.
- Tao3300 4y agoI've said it before, I'll say it again. I don't care if it's the Joint Chiefs and the whole United Nations calling me up to tell me an asteroid will destroy the planet unless I sign on [0]. If it's after hours, it'll have to wait until tomorrow, possibly Monday. [0] Spoiler alert: it's not.
- redfoxx22 4y ago
- Equiet 4y agoIt's surprising how seemingly difficult it is to build a good on-call scheduling system. Everything I tried so far (not naming the companies here) felt like the UX was the last thing on the developers' minds. Which is tolerable during business hours but really annoying at 2am. Is there some hidden complexity or is it just a consequence of engineers building a product for other engineers? Also, any tips what worked for you?
- matsemann 4y agoHave had lots of bad experiences with that from Pagerduty at least. Want to generate a schedule far in advance, so people know when they will be oncall and can plan/switch. Of course, in a few months we may have some new people having joined, some quit, or other circumstances. A single misclick when fixing that can invalidate the whole schedule and generate another. Infuriating. Or the UI itself, might have become better tha last two years, but having to click "next week" tens of times to see when I was scheduled (since I wasnt just interested in my next scheduled time but all of them) were annoying.
- deleted 4y ago[deleted]
- pphysch 4y agoA bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my current containerless setting.
- MarquesMa 4y agoThis. I find open source projects written in Go or Rust are usually more pleasant to work with than Java, Django or Rails, etc. They have less clunky dependencies, are less resource-hungry, and can ship with single executables which make people's life much easier. Just think about Gitea vs GitLab.
- matsemann 4y agoNot sure why you include java in that, as you mostly get a standalone file. No such thing as a jre in modern java deployment. As for python, at least getting a dockerfile helps a lot. Otherwise it's a huge mess to get running, yes. Python is still a hassle anyways, since the lack of true multithreading means that you often need multiple deployments, which the Celery usage here for instance shows.
- FridgeSeal 4y agoPython application deployments are all fun and games until suddenly the documentation starts unironically suggesting that you should “write your configuration as a Python script” that should get mounted to some random specific directory within the app as if that could ever be a sane and rational idea.
- Volundr 4y ago> Not sure why you include java in that, as you mostly get a standalone file. No such thing as a jre in modern java deployment. Maybe I'm behind the times, but I can't figure out what you mean here. As far as I know 'java -jar' or servlets are still the most common ways of running a Java app. Are you talking graal and native image?
- sandstrom 4y agoI think it would be great if it was easier to mix and match Grafana SaaS and self-hosted products. For example, we need to run Loki ourselves, for security / privacy reasons, but wouldn't mind using hosted versions of Tempo, Prometheus and OnCall. Right now it isn't super-easy to link e.g. self-hosted loki search queries with SaaS-Prometheus.
- netingle 4y agoIts very much our aim to make this mix of self-hosted and cloud services as easy as going all-cloud; but I agree we're not quite there yet. Do you mind if I ask what isn't super-easy about linking self-hosted loki search queries with SaaS-Prometheus? You should be e.g. able to add a Prometheus data source to your local Grafana (or securely expose your Loki to the internet and add a Loki data source to your Cloud Grafana)
- sandstrom 4y agoHonestly I haven't tried that much, but didn't find anything in the docs so I assumed it wasn't a prioritized area. In our particular scenario, we'd probably want to run Loki + Grafana locally, and then hosted Prometheus + hosted Grafana for metrics. But would be great if we could just tell the two about each other, and under which domains they exist. That way, Prometheus-grafana could construct URLs that linked straight into Loki-grafana (that we host) for e.g. the same interval, or the same label filter (GET params). But it would only work if I (the end-user) had access to both. That way, we don't have to expose Loki to the internet. But linking would still work. There are quite a lot of services that does this with Github and commits. You can link from e.g. Bugsnag to Github by only telling Bugsnag your org and repo names. But Bugsnag won't have read access to Github (they also have another integration method which does require access, but that's not the one I'm talking about here). Those types of "linking into a known URL pattern of another service" integrations are easy to setup and very easy to secure.
- pachico 4y agoI love Grafana, don't get me wrong, but I have the sensation they are now in that position where, companies that got a massive capital injection and, therefore, a massive increase of work power, release too much and too soon. It doesn't have anything to do, of course, with the fact that this morning we suddenly found that all our dashboards stopped working because we were upgraded to Grafana v9, for which there is not a stable release nor documentation for breaking changes. Luckily they rolled back our account.
- danlimerick 4y agoI apologize for the disruption we caused you when rolling out Grafana 9. We are working on improving our releases to Grafana Cloud and also on making sure that errors due to breaking changes in a major release won't affect customers in the future. As a Grafana Cloud customer, you shouldn't need to read docs about breaking changes when we upgrade your instance.
- pachico 4y agoDude, I hope you also read when I say that I love what you do and your reply just confirms I'm putting my money in the right hands. I just wouldn't mind to be the last to upgrade to a newer version :)
- greatgib 4y agoI would give a huge marketing bullshit award for the following sentence: <<We offered Grafana OnCall to users as a SaaS tool first for a few reasons. It’s a commonly shared belief that the more independent your on-call management system is, the better it will be for your entire operation. If something goes wrong, there will be a “designated survivor” outside of your infrastructure to help identify any issues. >> They tried to ensure that you use their SaaS offering because they care more about your own good than yourself. So humanist...
- ezrast 4y agoThe point isn't that their infrastructure is more reliable than yours, but that it's decoupled from yours. If you run your monitoring on the same infra as production, it's liable to go down when production does, i.e. just when you need it most. This is a real reason to outsource monitoring to a SaaS, just like there are real reasons to self-host. I mean, obviously they chose to address the segment of the market they could get more money out of first; I'm not contesting that. But the bit you quoted is low-grade bullshit at best. Hardly award-winning.
- Maledictus 4y agoWhat I really want is an Android app that keeps alerting until a page is ACKed or escalated.
- machinerychorus 4y agocheck out pushover, I use it for this exact case https://pushover.net/ https://pushover.net/
- ndom91 4y agoOpsGenie's Android app does this. Wouldn't be surprised if OnCall has this as well (or is coming soon).
- raffraffraff 4y agoProduction helm chart link on this page leads to 404: https://grafana.com/docs/grafana-cloud/oncall/open-source/#production-environment https://grafana.com/docs/grafana-cloud/oncall/open-source/#p...
- googletron 4y agoVery cool. I love what the Grafana team is up to.
- JimXugle 4y agoAnother similar tool I've used in the past is GoAlert. https://goalert.me/ https://goalert.me/
- NonNefarious 4y agoThe title is missing critical info: What the hell is it? Of course the article isn't much better. It reads like a joke, the joke being that "on-call management" doesn't mean anything.