9 ms·
Show HN: Dead man's switch without reliance on your infra
I wrote this Go project to implement a dead man’s switch that didn’t rely on cron jobs, timers, databases, etc on the infra it runs on. Deadcheck integrates to PagerDuty and keeps a long running incident snoozed until expected check-in times where it’ll alert unless a check-in occurs.
- hiatus 2y agoDo the people on the receiving end of a pagerduty notification require pagerduty accounts?
- adamdecaf 2y agoNo, you can get text, email, slack, etc alerts from PagerDuty without installing their app.
- 9dev 2y agoI’m not sure I understand what the application itself does; as far as I can see, it basically configures the external services to carry out the actual checks and ensures this configuration is up to date? Wouldn’t it be better to do this as part of a Terraform script or something? Not to take away from the project, it sure looks neat—just wondering if I really want to deploy yet another, independent IaC tool.
- hiatus 2y agoThere is a pagerduty provider for terraform, too. https://registry.terraform.io/providers/PagerDuty/pagerduty/latest/docs https://registry.terraform.io/providers/PagerDuty/pagerduty/...
- pigeonhole123 2y agoIt doesn't look like PagerDuty supports sending out alerts on missing heart beats
- hiatus 2y agoThat's exactly how escalation in pagerduty works.
- adamdecaf 2y agoThe idea is an app checks in after scheduled jobs, which extends the snooze
- bryanlarsen 2y agoAFAICT, that's a better description of the project for people like us. It's a way to add a missing heart beat alert to PagerDuty.
- adamdecaf 2y agoIt sets up expected check in times for an app/job. The job is responsible for checking in, but the alert will fire regardless of your infra. Doesn’t require a start command and your infra can completely fail and you’ll still get alerted.
- wil421 2y agoAny support for something besides pager duty? I work for a telco and there are so many different heartbeats we rely, this project seems very useful for something like that. The config file looks much better than the python/perl/bash whatever scripts we have scheduled on our systems.
- adamdecaf 2y agoYea I want to support other vendors. What do you use?
- lopkeny12ko 2y ago[flagged]
- zeroq 2y agoIt's a cloudification of dead man's switch.
- hypeatei 2y agoThat's what I thought; this tool is "kicking the can" to whatever third party service you're using. So, it's essentially just a tool that will ping stuff for you in a slightly more convienent way?
- adamdecaf 2y agoSystems always rely on something. PagerDuty has been very reliable for years and we use it for alerting, so relying on it more isn’t a big ask. I plan to support multiple integrations so you could get alerted from multiple streams.
- hooverd 2y agoYour infra is relying on an ISP and colo and the power company and maybe even the Post Office. You're always relying on something that's not you. Personally I'd rely on pre-paid AWS. If all of us-east-1 goes down for days then we probably have bigger problems.
- ellyagg 2y agoNow two isolated systems have to fail. Redundancy is fault tolerant. Nothing is fault proof.
- siliconc0w 2y agoA possible integration are messaging apps that support send later. I sometimes set these up if I'm going hiking.
- adamdecaf 2y agoYea, and you would delete the pending message. Good idea…
- herpderperator 2y agoSo... like BetterStack's heartbeat monitor? [0] [0] https://betterstack.com/docs/uptime/cron-and-heartbeat-monitor/ https://betterstack.com/docs/uptime/cron-and-heartbeat-monit...
- tnolet 2y agoor Checkly's heartbeat check https://www.checklyhq.com/docs/heartbeat-checks/ https://www.checklyhq.com/docs/heartbeat-checks/
- adamdecaf 2y agoI don't see how to configure anything but an interval (e.g. every N minutes) in checklyhq. Deadcheck allows you to expect check-ins at arbitrary times (e.g. 9am, 1:15pm, 3:15pm on weekdays). https://developers.checklyhq.com/reference/postv1checksheartbeat https://developers.checklyhq.com/reference/postv1checksheart...
- neilv 2y agoBetter Stack got on my disapproval list, for spamming me with a series of automated sales emails 15-20 minutes apart, which seemed designed that way. "One more thing.", etc. They should've sent one welcome&upsell email, not several. And not abused contact info that was intended to reach me reliably and promptly. For my immediate very simple need, Uptrends didn't spam me, and so far they've reliably and promptly notified me whenever my site is down even briefly (e.g., rebooting for kernel update), so they'll be getting money from me as I grow. https://www.uptrends.com/ https://www.uptrends.com/
- herpderperator 2y agoSorry to hear. I didn't have that experinece; it's been pleasant for me (there was a billing snafu while they changed their plan structure but it was resolved with a support email) and their monitoring has been very reliable since I signed up a year ago. Their service also seems to be better (4 checks/locations vs Uptrend's 2.) Anyway, tl;dr is I never had an issue that resulted in not getting alerted about something that I needed to be alerted about, which, above all else, is the point of their offering.
- rozenmd 2y agoThe term of art you're looking for is: "heartbeat check"/"healthcheck", or most commonly: "cron job monitor".
- herpderperator 2y agoYeah I thought "dead man's switch" was such a weird name for this.
- LorenPechtel 2y agoIt's something that happens if you fail to take action to prevent it at some specified frequency. Looks a lot like a dead man switch to me.
- adamdecaf 2y agoThere’s no crontab for banking days. Deadcheck also requires checkins within a delta of the expected checking time. Deadcheck also doesn’t rely on your infra to alert.
- bravetraveler 2y agoSeems very similar to a 'watchdog', just... reporting instead of doing anything about it.
- vivzkestrel 2y agouptime monitor?
- rozenmd 2y agoUptime monitors ping your server, this seems to do the opposite.
- turtlebits 2y agoI'm not sure running an additional service plus a SaaS requirement is better than relying on infra. Standing up this service is going to rely on your infra. What if it goes down? If you're going to have to write a scheduled check-in anyways, why not use something like cronitor and reduce the complexity? (they host the check-in endpoint)
- psnehanshu 2y agoIf the infra goes down, you fix it while alive, otherwise the snooze timeout on the PagerDuty incident will expire and that will trigger the configured tasks. If PagerDuty goes out of business before that, then that's a different discussion.
- adamdecaf 2y agoExactly. At Moov we rely on PD, so if they’re down we have bigger issues anyway. I plan to support additional integrations so a check-in could update multiple
- turtlebits 2y agoMy point was - if you're going to rely on a SaaS (Pagerduty), why not just use one that includes health check monitoring (ie cronitor) and cut out this self hosted webhook service?
- adamdecaf 2y agoDeadcheck handles calculating the snooze durations for you. It could be a library as well.
- remram 2y agoIf you call this a "dead man's switch" I'd expect it to turn my app off when I die. E.g. "switch" something when something happens to a "man". Your own README links to this definition: "A dead man's switch is a switch that is designed to be activated or deactivated if the human operator becomes incapacitated". This is a watchdog timer / monitor / heartbeat, setting off an alert if a timer elapses.
- aftbit 2y agoThe "original" dead man switch as I heard about it was a pedal on a train that would apply the brakes if the operator released it. I've often wondered about how to reliably take software actions after my death or dishonor. After all, you can't really rely on me being able to pay my bills. I'm not looking to do something expensive, more like delete my accounts and send some messages.
- DowagerDave 2y ago>> after my death or dishonor honest question: why do you care?
- vincnetas 2y agoif you care about people that you leave behind...
- nurettin 2y agoFrom experience, it is a huge inconvenience to people surviving the deceased leaving without any way to pay their debits. And if you don't care about what happens after you die, why did you even care when alive? Why not always be a dick? At least it is consistent.
- hotspot_one 2y agoUnder US law, the debts die with the person. You are under no obligation to repay your parent's debts. Now if the debt is tied to a house (mortgage) or a car (car loan), you might lose the house/car if you don't pay, but you do not have an obligation to pay. Likewise failure to pay will not impact your credit. So if I die in debt up to my eyeballs, and if I am sole signatory on those debts, I have only hurt my creditors, not my family. caveats-- if my family was counting on the house and I have an unaffordable mortgage, then yes I have caused them harm. Likewise other irresponsible debts. -- at the end of the chain, creditors are also people. It is their job to loan money at risk, so their loss is their problem, but this assumes I was dealing in good faith when I took the loan.
- jbverschoor 2y agoNot to be confused with deadmanssnitch.com
- roshan8 2y agoNice one! Do you have any plans to support something else apart from Pager duty. I have been using Squadcast.com for my hobby project monitoring. Integration with that would be nice. I'm open to create a PR if you are interested.
- adamdecaf 2y agoThanks, yea I’d like to support other vendors. Haven’t heard of Squadcast I have https://github.com/adamdecaf/deadcheck/issues/12 https://github.com/adamdecaf/deadcheck/issues/12 tracking additional vendors Edit: I'm open to a PR if you're willing. Curious, but would you use their delayed notification config to implement? https://apidocs.squadcast.com/?version=latest#7742a9af-29fe-4c08-b7c6-40a1f1796e2f https://apidocs.squadcast.com/?version=latest#7742a9af-29fe-...
- focusedone 2y agoI feel like posting a software project to HN and meeting criticism is some near-end-stage of programming mastery.
- wcallahan 2y agoWas thinking the same!
- divbzero 2y agoYes, it’s a positive signal when people take it seriously enough to criticize.
- arisAlexis 2y agoActually I got fried and very bad remarks on my project. Not true that hn crowd is friendly to show hn projects at all.
- encoderer 2y agoCool trick using PagerDuty like that. If you don’t want to use a service like Cronitor, you can self-host this without the usual fear that an outage will also take down your monitoring.
- adamdecaf 2y agoYep. I looked at Cronitor and thought about using it, but direct to PD removed a step for us at Moov.
- EGreg 2y agoSpeaking of not using anyone’s specific infra, we deployed software on blockchains to do this. We had to implement a “heartbeat” and “succession” in our blockchain-based solutions for organizations to control things together. It’s part of our “application suite for organizations” where each one is sort of this general-purpose LEGO block that could be used to build a custom solution. In each case, you’d enter some parameters and create an instance from a Factory. We made it simple and secure for any organization to use this. In this specific one, ControlFactory is used to create ControlContract instance that can be used to control an address together (to sign off things collectively, like transfer tokens or call an arbitrary method on a different address) We had to handle what happens if the M of N people don’t show up for a while. And we said they have to call a heartbeat() method every so often. If they fail to call it then control temporarily passes to the next group in succession, until the OGs can finally call the heartbeat() method again. Here is more info on the why: https://community.intercoin.app/t/intercoin-applications-control-and-multisignature/2898 https://community.intercoin.app/t/intercoin-applications-con... And here are the rest of the blockchain apps for organizations: https://community.intercoin.app/t/applications-of-intercoin-making-crypto-mainstream/87 https://community.intercoin.app/t/applications-of-intercoin-... You can go ahead and use it, the factory it’s been deployed on many EVM blockchains, at the same address. PS: fun fact, you can configure a ControlContract to also manage calling methods on itself, thereby creating custom “policies” for organizations when it comes to granting/recoving rights of other people to the quorum.
- joshbetz 2y agoVery clever. I have Alertmanager in a second region so it can check on my first Alertmanager, but this is much nicer.
- grahamj 2y agoNeat. We use OpsGenie to monitor our alerting infra and it has deadman/heartbeat support so something like that is another way to go.
- adamdecaf 2y agoHow are they triggering the alert when the dead man’s switch isn’t tripped in time?
- grahamj 2y agoIt's basically an alerting platform so it's just what they do. It integrates with our alerting for things like metrics monitoring and it'll route the notifications based on on-call schedule etc. Deadman is just another way to trigger.
- cduzz 2y agoI guess I'm an oldster shaking my fist at clouds, but ... Depending on where you live, and if it matters, give it to an attorney. Maybe use this to trigger the notification to your attorney, with instructions on how to double check things before doing things.
- danbruc 2y agoI agree, I think people underestimate the importance of actually checking why the heart beat was missed, double checking is important. You might think you can easily click some button once a week and it is no problem to take some drastic and maybe irreversible actions after a week without any event. And then you get into a serious accident and end up in hospital for weeks. Or there is some bug unintentionally triggering the actions.
- cuu508 2y agoThe configuration example in README has "Reports Finalized", so I assume this is more for IT infra monitoring, than for taking irreversible actions when you become incapacitated :-)
- xyst 2y agoI’ll use this for my next wormable vuln instead of hard coding DNS checks ;)
- christina97 2y agoI sometimes think about a “dead man service”: you leave instructions (that you upload to the service), then when you pass away/etc, the service operators go and follow your instructions and do whatever you asked for. You’d pay some pre-agreed sum, possibly annuity-type subscription, and at the end we go and follow your wishes. Basically a technically competent will executor. It’s probably too much to expect your family to know how to operate your systems. Maybe you encrypt the instructions and give fractional keys to family etc.
- INTPenis 2y agoI've thought about this too but I think the type of business best suited to execute this is a law firm, or you know, the normal executor of people's wills. They just need the IT bit.
- lancesells 2y agoThis is an attorney. No subscription, no tech, no bugs, no encryption, less hackable.
- mhink 2y ago> Basically a technically competent will executor. I don't think OP is saying that an attorney *can't* get these things done, but that it would make them feel more comfortable knowing that a technically competent person and/or service will be performing the actual actions. I do think there's a place for an attorney here, in the sense that they could be the trusted individual responsible for notifying DeadManService, Inc. that a particular person has, indeed, passed on and wishes DeadManService to run their instructions.
- ufmace 2y agoI'm not sure what value adding a DeadManService would have then. Probably simpler to just have the attorney's instructions say, "Hire a technical consultant to carry out the following:".
- INTPenis 2y agoI did something similar years ago when I was working with observability. Prometheus alertmanager triggers a special alert constantly that calls a lambda (or any webhook), so when alertmanager dies or is unable to alert then the lambda will send an alert over a 3rd party push service to notify ops that alertmanager is down. We called it a dead man's switch but it was really just a way to monitor alertmanager.
- adamdecaf 2y agoYea I’ve setup two alertmanagers that check each other before. It’s useful for multi site deployments.
- v3ss0n 2y agoBetter have something that runs off the cloud. PagerDuty is not something under your countrol. A stuff as imporant as dead man switch should not rely on someone's service.
- tatersolid 2y agoOkay, so now your dead man’s switch depends on your home or colo’s power, internet connection, compute hardware, storage hardware, network hardware, software stack maintenance, plus the ability to pay bills for those on time? Pre-Paying a cloud service pennies per month to do all that for you will likely be more reliable and much simpler.
- PeterisP 2y agoI'd argue entirely opposite, that a dead man's switch should 100% rely only on someone else's services and should avoid any dependencies on your stuff, as something happening to you is correlated to something happening to things you run or own, and the whole point of such a switch is that it should function properly when e.g. all your stuff burns down or gets confiscated or whatever.
- ForHackernews 2y agoI don't know about the rest of the people here, but I aim to outlive PagerDuty.
- arisAlexis 2y agoThe problem with these kind of project is that they are no good for very serious situations. Say Navalny set it up. They kidnap someone, the make you delete the pagerduty service and then kill you. Immutable / undeletable tasks would be needed.