6 ms·

Using AWS Lambda to call and text you when your servers are down

160 points by nhm 10y ago

Dobbs 10y ago
From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph. Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your outage is that AWS is having issues. Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant.
- nhm 10y ago
  > outsource to people who obsess over doing this right Completely agree! I often have to fight that "I could just build that myself" mentality, which glosses over the points you made so well.
  - avitzurel 10y ago
    This and that! It's the same as "Twitter clone" with just posting messages with 140 char limit and "build a blog in 15 minutes". Alerting over a downed website are is sorta like a glacier, there's so much under the surface, if you just see the surface you're missing out. 1. Multiple locations 2. Multiple check intervals 3. SMS/email provider switch on fail 4. Auto recovery of your checkers 5. Multiple providers with a single storage.
- tjholowaychuk 10y ago
  Yep, plus most engineering time is worth at minimum $60+/h, which would pay for a year or more with most of these services.
  - vacri 10y ago
    On the other hand, it's 'set up once and it just keeps chugging along', and isn't Yet Another SaaS To Manage. Also, if you want a 'proper' ops alerting SaaS, you're looking at something along the lines of $50/user/mo or $15/server/mo, neither of which is trivial.
    - tjholowaychuk 10y ago
      Yeah assuming nothing falls apart with the custom implementation maintenance-wise. Programmers have a hard time focusing on their real goals though, we often re-implement things that really aren't worth the time or money.
- IgorPartola 10y ago
  My favorite issue recently came up with a Django app of mine which was set up to email me when a request errors out. Turns out, when I switched which server it ran on I misconfigured the email settings and one of the errors was caused due to the inability to send an email. Thankfully it only took a few days to figure this out.
- melvinmt 10y ago
  > Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant. You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-pingdom http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...
  - user5994461 10y ago
    Chances are that pingdom won't be down at the same time that your site is down. Diversify to avoid cascading failures ;)
  - imtringued 10y ago
    With your own solution you will likely encounter the same problems that pingdom faced including this one. The benefit of a service like pingdom is that they already solved those problems for you or if they haven't you don't have to waste time solving them yourself. It's not very efficient if everyone solves the same problems over and over again.
  - dharma1 10y ago
    Use 2 or more providers. Signing up takes a minute or two and there are free alternatives
- balaa 10y ago
  HN top comments are getting way too predictable.
- teddyh 10y ago
  We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being down. A huge pain.
  - snom380 10y ago
    Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.
    - teddyh 10y ago
      If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.
      - dalore 10y ago
        Yes you were down for some of your users. If that's ok for you that's fine. But if I were you I would be calling my ISP and trying to sort out why customers from location X can't access but customers from location Y can. If you're providing a service to your users, and they say that the service is down using pingdom, you should be looking into, not just saying "Works on my machine".
        teddyh 10y ago
        Why should we be the ones to look into it? It was a random intermittent short-duration fault in the middle if the Internet, at some unknown place on the then-current path between us and Pingdom. Why should not Pingdom be at least equally as obligated to look into it? After all, they’re the ones actually using the failing connection, in order to monitor our and others’ services. But no, Pingdom simply report us as being down, and leave the hard part to us; i.e. the part where we have to explain to our customers that the Pingdom report is actually provably incorrect. I mean, what qualifies as “being up”? If some random link in the middle of the Internet goes down, and you suddenly, for 30 seconds, are unreachable for the few hundred people going through that exact link because it happens to be the best path between those people and your server, can they claim that you have failed to provide adequate uptime? If such a fault happens, are you then responsible to troubleshoot it? I say no. The Internet is the ISP’s responsibility, and the only faults actually meaningful to report to your ISP are the repeatable or long-lasting ones. Small stuff like this is not worth anybody’s time (except ISPs) to go digging into.
falcolas 10y ago
And I'm sure Lambda will never go down. Right? Right?? (It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)
- jlgaddis 10y ago
  Well, sure, of course it will, but I don't think Nick is advocating replacing a complete, full featured monitoring system with this. It could be very useful to, for example, keep an eye on your monitoring system. At $work, we have a pretty extensive monitoring system that we've built out. We use an external service to watch over the monitoring system, though, to alert us of any issues with it that we haven't otherwise caught. Besides, like he said, it's "fun" and kinda neat.
- cddotdotslash 10y ago
  Of course it can go down, and you can have CloudWatch alerts to alert you about that. But so can your Nagios server sending pings go down or the fancy SaaS you signed up for.
  - robinson-wall 10y ago
    Did you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?
    - cddotdotslash 10y ago
      Yes, because they're different services running on different architecture and distributed differently. I challenge you to find one time in the past five years where CloudWatch was down at the same time as other services. Even if you can, I'm sure your custom built Nagios server in your datacenter has gone down as many or more times too. But my bigger point here is that you're essentially asking "well how do you monitor your monitor?" At which point up the chain do you have enough? Also, I think the original post was simply a demo of what is possible. Yet whenever someone posts something, people go in the comments to belittle it. "Yeah, you built a monitoring solution... Well what happens if that goes down?" Which is a legitimate question. But obviously if your production service is that critical to your business, you won't be monitoring it with a service that costs $0.0000002 per execution.
tymm 10y ago
I wrote something similar in bash and put it into a docker image: https://hub.docker.com/r/simplepush/alerta https://hub.docker.com/r/simplepush/alerta Just running this docker image on a server you want to monitor is enough. Instead of Twilio it uses Simplepush (https://simplepush.io https://simplepush.io).
- cyberferret 10y ago
  Simplepush looks like a cool service - thanks for the heads up. It seems that it accomplishes the author's main need - that for a constant buzzing which needs to be picked up and dealt with. EDIT: Just seen that it is Android only! :-/
  - ubercow 10y ago
    If you need something similar that works on iOS (and Android), take a look at https://pushover.net/ https://pushover.net/ I use it for some personal automation scripts that might need to get my attention if something goes wrong.
tjholowaychuk 10y ago
I wrote Apex Ping (https://apex.sh/ping/ https://apex.sh/ping/) for those who want more features and/or don't want to waste the time to save a few bucks :D.
- gingerlime 10y ago
  Apex ping is great, but I'm still waiting for SMS / Twilio integration (hint, hint, nudge nudge) :)
  - tjholowaychuk 10y ago
    :D I wish SMS wasn't so awkward, you're pretty much forced to have a credit system since it's so expensive. I'll probably still do it at some point. Makes it awkward for the customer as well if you have to babysit the credits
    - gingerlime 10y ago
      I suggested it before, but I think you can work around it: Let your customers give you their Twilio API key (with a big disclaimer that any charges by Twilio are not your responsibility...).
      - tjholowaychuk 10y ago
        If I can get costs down elsewhere I'll maybe just do "unlimited" on some of the accounts, where "unlimited" is some large arbitrary number haha.
        cuu508 10y ago
        That would be the nicest user experience for your users, but it is a bit risky. You probably have a "reasonable number of notifications per user per month" in mind. As you sign up new users, you will sooner or later get some that will exceed that number by a lot--without a malicious intent.
    - 10y ago
intrasight 10y ago
The advantage of this type of cloud solution over a one-size-fits-all cloud service like Pingdom (which I use) is flexibility. You can configure cloud agents to perform nearly any task you can envision.
gravypod 10y ago
I don't know if everyone knows this but you can make texts using email. Most providers have SMTP gateways for SMS services. Verizon runs @vtext.com
- gtaylor 10y ago
  Just keep in mind that these aren't incredibly reliable across the board. Others have very low or arbitrary autoban or blacklist policies. I eventually caved and paid Twilio to hassle with SMS logistics for me, rather than deal with the weirdness.
- tomascot 10y ago
  I did this for non-critical ops, it's just a foward to my phone's email address, really simple. The problem is that not every company is reliable or even has that email
social_quotient 10y ago
Note: instead of dynamodb for the lookup mentioned at the bottom, maybe consider https://aws.amazon.com/athena/ https://aws.amazon.com/athena/ for an s3 query
- illumin8 10y ago
  You could literally just store a .CSV file in S3 with a table that has the on-call schedule in it and run SQL queries against Athena that would be cheap... you'd be querying a few KB, but DynamoDB is probably better for this use case, honestly. Athena is great for scanning huge datasets very quickly.
dmourati 10y ago
Isn't the better plan to use Lambda instead of your servers?
theparanoid 10y ago
I've used montastic.com for years. 2min setup to fire and forget.
justinc8687 10y ago
I use https://aremysitesup.com/ https://aremysitesup.com/ and I've found it really helpful as it one of the few inexpensive services I've found that will CALL me if things are down. SMS is nice, but I use the do-not-disturb feature on my phone in the evenings, and at least on iOS, the only way to punch through that is with a call from a number on my favorites list. This meets that need very well and I've found the service to be quite spot on alerting me (both when I had one instance of things hitting the fan, but also during scheduled maintenance). I'd highly recommend.
cagataygurturk 10y ago
Route53 Health Checks & SNS can send sms message without any Lambda involved.

160 points by nhm 10y ago

Dobbs 10y ago

From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph. Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your outage is that AWS is having issues. Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant.

nhm 10y ago
> outsource to people who obsess over doing this right Completely agree! I often have to fight that "I could just build that myself" mentality, which glosses over the points you made so well.
- avitzurel 10y ago
  This and that! It's the same as "Twitter clone" with just posting messages with 140 char limit and "build a blog in 15 minutes". Alerting over a downed website are is sorta like a glacier, there's so much under the surface, if you just see the surface you're missing out. 1. Multiple locations 2. Multiple check intervals 3. SMS/email provider switch on fail 4. Auto recovery of your checkers 5. Multiple providers with a single storage.
tjholowaychuk 10y ago
Yep, plus most engineering time is worth at minimum $60+/h, which would pay for a year or more with most of these services.
- vacri 10y ago
  On the other hand, it's 'set up once and it just keeps chugging along', and isn't Yet Another SaaS To Manage. Also, if you want a 'proper' ops alerting SaaS, you're looking at something along the lines of $50/user/mo or $15/server/mo, neither of which is trivial.
  - tjholowaychuk 10y ago
    Yeah assuming nothing falls apart with the custom implementation maintenance-wise. Programmers have a hard time focusing on their real goals though, we often re-implement things that really aren't worth the time or money.
IgorPartola 10y ago
My favorite issue recently came up with a Django app of mine which was set up to email me when a request errors out. Turns out, when I switched which server it ran on I misconfigured the email settings and one of the errors was caused due to the inability to send an email. Thankfully it only took a few days to figure this out.
melvinmt 10y ago
> Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant. You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-pingdom http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...
- user5994461 10y ago
  Chances are that pingdom won't be down at the same time that your site is down. Diversify to avoid cascading failures ;)
- imtringued 10y ago
  With your own solution you will likely encounter the same problems that pingdom faced including this one. The benefit of a service like pingdom is that they already solved those problems for you or if they haven't you don't have to waste time solving them yourself. It's not very efficient if everyone solves the same problems over and over again.
- dharma1 10y ago
  Use 2 or more providers. Signing up takes a minute or two and there are free alternatives
balaa 10y ago
HN top comments are getting way too predictable.
teddyh 10y ago
We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being down. A huge pain.
- snom380 10y ago
  Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.
  - teddyh 10y ago
    If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.
    - dalore 10y ago
      Yes you were down for some of your users. If that's ok for you that's fine. But if I were you I would be calling my ISP and trying to sort out why customers from location X can't access but customers from location Y can. If you're providing a service to your users, and they say that the service is down using pingdom, you should be looking into, not just saying "Works on my machine".
      - teddyh 10y ago
        Why should we be the ones to look into it? It was a random intermittent short-duration fault in the middle if the Internet, at some unknown place on the then-current path between us and Pingdom. Why should not Pingdom be at least equally as obligated to look into it? After all, they’re the ones actually using the failing connection, in order to monitor our and others’ services. But no, Pingdom simply report us as being down, and leave the hard part to us; i.e. the part where we have to explain to our customers that the Pingdom report is actually provably incorrect. I mean, what qualifies as “being up”? If some random link in the middle of the Internet goes down, and you suddenly, for 30 seconds, are unreachable for the few hundred people going through that exact link because it happens to be the best path between those people and your server, can they claim that you have failed to provide adequate uptime? If such a fault happens, are you then responsible to troubleshoot it? I say no. The Internet is the ISP’s responsibility, and the only faults actually meaningful to report to your ISP are the repeatable or long-lasting ones. Small stuff like this is not worth anybody’s time (except ISPs) to go digging into.

falcolas 10y ago

And I'm sure Lambda will never go down. Right? Right?? (It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)

jlgaddis 10y ago
Well, sure, of course it will, but I don't think Nick is advocating replacing a complete, full featured monitoring system with this. It could be very useful to, for example, keep an eye on your monitoring system. At $work, we have a pretty extensive monitoring system that we've built out. We use an external service to watch over the monitoring system, though, to alert us of any issues with it that we haven't otherwise caught. Besides, like he said, it's "fun" and kinda neat.
cddotdotslash 10y ago
Of course it can go down, and you can have CloudWatch alerts to alert you about that. But so can your Nagios server sending pings go down or the fancy SaaS you signed up for.
- robinson-wall 10y ago
  Did you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?
  - cddotdotslash 10y ago
    Yes, because they're different services running on different architecture and distributed differently. I challenge you to find one time in the past five years where CloudWatch was down at the same time as other services. Even if you can, I'm sure your custom built Nagios server in your datacenter has gone down as many or more times too. But my bigger point here is that you're essentially asking "well how do you monitor your monitor?" At which point up the chain do you have enough? Also, I think the original post was simply a demo of what is possible. Yet whenever someone posts something, people go in the comments to belittle it. "Yeah, you built a monitoring solution... Well what happens if that goes down?" Which is a legitimate question. But obviously if your production service is that critical to your business, you won't be monitoring it with a service that costs $0.0000002 per execution.

tymm 10y ago

I wrote something similar in bash and put it into a docker image: https://hub.docker.com/r/simplepush/alerta https://hub.docker.com/r/simplepush/alerta Just running this docker image on a server you want to monitor is enough. Instead of Twilio it uses Simplepush (https://simplepush.io https://simplepush.io).

cyberferret 10y ago
Simplepush looks like a cool service - thanks for the heads up. It seems that it accomplishes the author's main need - that for a constant buzzing which needs to be picked up and dealt with. EDIT: Just seen that it is Android only! :-/
- ubercow 10y ago
  If you need something similar that works on iOS (and Android), take a look at https://pushover.net/ https://pushover.net/ I use it for some personal automation scripts that might need to get my attention if something goes wrong.

tjholowaychuk 10y ago

I wrote Apex Ping (https://apex.sh/ping/ https://apex.sh/ping/) for those who want more features and/or don't want to waste the time to save a few bucks :D.

gingerlime 10y ago
Apex ping is great, but I'm still waiting for SMS / Twilio integration (hint, hint, nudge nudge) :)
- tjholowaychuk 10y ago
  :D I wish SMS wasn't so awkward, you're pretty much forced to have a credit system since it's so expensive. I'll probably still do it at some point. Makes it awkward for the customer as well if you have to babysit the credits
  - gingerlime 10y ago
    I suggested it before, but I think you can work around it: Let your customers give you their Twilio API key (with a big disclaimer that any charges by Twilio are not your responsibility...).
    - tjholowaychuk 10y ago
      If I can get costs down elsewhere I'll maybe just do "unlimited" on some of the accounts, where "unlimited" is some large arbitrary number haha.
      - cuu508 10y ago
        That would be the nicest user experience for your users, but it is a bit risky. You probably have a "reasonable number of notifications per user per month" in mind. As you sign up new users, you will sooner or later get some that will exceed that number by a lot--without a malicious intent.
  - 10y ago

intrasight 10y ago

The advantage of this type of cloud solution over a one-size-fits-all cloud service like Pingdom (which I use) is flexibility. You can configure cloud agents to perform nearly any task you can envision.

gravypod 10y ago

I don't know if everyone knows this but you can make texts using email. Most providers have SMTP gateways for SMS services. Verizon runs @vtext.com

gtaylor 10y ago
Just keep in mind that these aren't incredibly reliable across the board. Others have very low or arbitrary autoban or blacklist policies. I eventually caved and paid Twilio to hassle with SMS logistics for me, rather than deal with the weirdness.
tomascot 10y ago
I did this for non-critical ops, it's just a foward to my phone's email address, really simple. The problem is that not every company is reliable or even has that email

social_quotient 10y ago

Note: instead of dynamodb for the lookup mentioned at the bottom, maybe consider https://aws.amazon.com/athena/ https://aws.amazon.com/athena/ for an s3 query

illumin8 10y ago
You could literally just store a .CSV file in S3 with a table that has the on-call schedule in it and run SQL queries against Athena that would be cheap... you'd be querying a few KB, but DynamoDB is probably better for this use case, honestly. Athena is great for scanning huge datasets very quickly.

dmourati 10y ago

Isn't the better plan to use Lambda instead of your servers?

theparanoid 10y ago

I've used montastic.com for years. 2min setup to fire and forget.

justinc8687 10y ago

I use https://aremysitesup.com/ https://aremysitesup.com/ and I've found it really helpful as it one of the few inexpensive services I've found that will CALL me if things are down. SMS is nice, but I use the do-not-disturb feature on my phone in the evenings, and at least on iOS, the only way to punch through that is with a call from a number on my favorites list. This meets that need very well and I've found the service to be quite spot on alerting me (both when I had one instance of things hitting the fan, but also during scheduled maintenance). I'd highly recommend.

cagataygurturk 10y ago

Route53 Health Checks & SNS can send sms message without any Lambda involved.