6 ms·

Tarsnap performance issues in late March, most of April

172 points by pndmnm 11y ago

mtsmith85 11y ago
This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems healthy or reassuring that incredibly smart people like Colin Percival suffer from similar challenges around fully understanding the scope of the problem and the solution. All that being said, I really respect the detailed response from a technical perspective as well as owning up to (and the decisions that went into) a spell of downgraded performance. Later edit because I don't want to spam the comments: I'd love some context (maybe from cperciva himself?) around the performance enhancement of integrating new Intel AESNI instructions. This is well beyond my depth and while Colin mentions that it didn't necessarily increase performance, I'm wondering if the hope is it would longterm? Or were there other benefits to such an integration?
- Poiesis 11y ago
  I came here to write this comment essentially. Very much of the time I feel, "If I knew what the problem[s] [was|were] it'd be solved by now!" That's not exactly true of course but of course diagnosis is a large part of the total solution. This type of an answer that Colin gave above does not exactly win friends and influence people in most situations where you're part of a team or hierarchy. Can anyone share what they've done to give better answers in these cases? I understand why people want the answers, but I don't have them to give right away particularly when it's Someone Else's system.
  - cperciva 11y ago
    Just to be clear, when Tarsnap users wrote to me I told them everything I could. The "I think it will be fixed soon" delay in sending out an email to the lists affected only people who didn't notice or noticed but didn't ask about the issue.
  - jballanc 11y ago
    One trick that I've learned (though I still have trouble routinely applying it myself) for these situations is: less is more. That is, as engineers we tend to want details. All the details. We want to know what happened, why it happened, how it's going to be fixed, and how long that will take. Because we want all that detail for ourselves, we hesitate to contact our customers/boss until we have all the details. Combine that with a desire to fix problems as they come up, and you end up with, "I never told you there was a problem because I was always one fix away from the solution." But most people are not engineers. They want to be acknowledged. They want to feel informed, even if they have less details than what you would like to provide for them. Sometimes, something as simple as, "We've noticed that there is an issue and are currently working on a fix," goes a long way. Also don't be afraid to pull out, "Users have been reporting issues with backup performance. We do not currently believe this represents a service failure, but we are working to return performance to normal levels." Your users trust you (otherwise they wouldn't pay you). If you "believe" something, they will too.
    - annnnd 11y ago
      This. There is also a saying on a similar note: "It's OK to disappoint, it's not OK to surprise."
      - psykovsky 11y ago
        What if the disappointment is a surprise?
- cperciva 11y ago
  I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed This ties in to the last lesson I mentioned at the bottom: 5. When performance drops, it's not always due to a single problem; sometimes there are multiple interacting bottlenecks. Every time I identified a problem, I was correct that it was a problem -- my failing was in not realizing that there were several things going on at once.
  - mtsmith85 11y ago
    That totally jibes with what I found "reassuring" in a sense. That even very smart people sometimes get hit with inadvertent "multiple problems looking like a single issue" situations.
  - jcrites 11y ago
    > Every time I identified a problem, I was correct that it was a problem -- my failing was in not realizing that there were several things going on at once. Very common! One thing that's been helpful for us is establishing predefined system performance thresholds that, if exceeded, initiate the chain of events that will lead to customer communication. "If X% of requests are failing, then we had better advertise that the system is degraded." Discussing and setting these thresholds in advance and the expectation that they'll result in communication helps drive the right outcome. It's not perfect, because one is always tempted to make a judgment call in the circumstance, which is vulnerable to the same effect, but it's a good start. Thanks for sharing!
  - spydum 11y ago
    i tend to get to debug problems like this (usually in 3rd party code i dont know the internals of) pretty frequently.. my experience has been it tends to follow a curve..MOST of the time, the problem is simple and you can quickly dispatch it. the scary (or fun, depending on your perspective) part hits when you pass the first level, and there are still problems.. and you dont know if it's two or ten levels deeper. then you get into that crazy test/optimize cycle and crawl out two weeks later wondering when you last ate..
- cperciva 11y ago
  I'd love some context (maybe from cperciva himself?) around the performance enhancement of integrating new Intel AESNI instructions. I was using OpenSSL for that (which was using a software implementation). The code (you can see it in spiped) now detects the CPU feature and selects between AESNI or OpenSSL automatically. Given that the tarsnap server code was spending about 40% of its time running AES, it's a nontrivial CPU time saving. I should probably have been clearer in my writeup though -- using AESNI was never a "once I roll this out everything will be good" fix. Rather, it was a case of "I have this well-tested code available which will help a bit while I finish testing the real fixes".
  - gonzo 11y ago
    One wonders why you aren't using a version of OpenSSL that has the AESNI bits already in it.
    - cperciva 11y ago
      I've learned that wondering about OpenSSL internals is detrimental to my sanity.
- mryan 11y ago
  This "it's almost fixed, I'll email the client soon" pattern is something I have personally struggled with a lot, and I agree it appears to be common with engineers. My workaround has been to make something else responsible for sending the email. In a team, this could be a manager setting a cut-off point after which communication must be made. When working on my own, I set an alarm for X minutes. When that alarm goes off I ignore the internal voice which says "just try one more thing, then send the email", and send an update to let the relevant people know my current progress, ETA to fix, and when they can expect the next update. I think this is similar to how GTD encourages us to use systems for storing to-do lists instead of trying to remember them - our fragile human brains are not always to be trusted.
patio11 11y ago
In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't trip our monitoring at the time because they all completed successfully. n.b. Our backups run outside of the hotspot times for Tarsnap, so we may have had less performance impact than many customers. I have an old habit of "Schedule all cron jobs to start predictably but at a random offset from the hour to avoid stampeding any previously undiscovered SPOFs." That's one of the Old Wizened Graybeard habits that I picked up from one of the senior engineers at my last real job, which I impart onto y'all for the same reason he imparted it onto me: it costs you nothing and will save you grief some day far in the future.
- mtsmith85 11y ago
  Hear hear on said Old Wizened Graybeard habit. The amount of pain inflicted from twenty jobs all starting up at :00 (or even :30, :45, etc.) when they could easily run at :04 or :17 can be huge. Anecdotally I once "lost" a sandbox server to a ton of developer sandbox jobs starting at :00 and not completing before the next batch started.
  - protomyth 11y ago
    Funny part to that, was on a project with multiple teams with multiple crontabs. Each team took that advice to heart for some jobs. Sadly, we had too many Hitchhiker fans and :42 became a bit too common.
    - kijin 11y ago
      Use the following shell command to decide when to run cron jobs. echo $((RANDOM % 60)) It's not a CSPRNG, but good enough for this kind of load balancing!
      - 11y ago
deleted 11y ago
[deleted]
appsonify 11y ago
what the fu....Colin Percival used to be my cello teacher 12 years ago....and he is running tarsnap. My mind is blown.
- cperciva 11y ago
  Not me -- I'm a violinist, and I've never taught anyone violin either. Maybe you're thinking of my bother (Graham)? He was teaching cello around that time period I think.
  - appsonify 11y ago
    Oh wow. Yes! Now I remember it was Graham. I was his student around then. Saw your photo on twitter and looked exactly like him! Hahaha. How is Graham? My mind is just completely blown right now.
    - cperciva 11y ago
      Graham is in Japan right now, but returning to Canada soon; he'll be joining me in the West Coast Symphony for our June concert. I don't feel that I should really be talking too much about my family in a public forum, but if you'd like to send me an email I can forward it to him.
ac29 11y ago
Sorry if this is offtopic, but can anybody explain the value proposition of tarsnap to me? It seems like a nice service and all, but the pricing is an order of magnitude more expensive than S3. If you are storing a few GB, this might not matter ("over half of Tarsnap users spend under $1 per month on storing their backups"), but if you have that little data, why not just dump it on a free Dropbox/Gdrive/etc account? For more data, why not just use one of the many compressed, deduplicated, encrypted, incremental backup systems (attic comes to mind, I'm sure there are others) then just sync to S3 at a tenth the cost?
- segf4ult 11y ago
  Because tarsnap is cheap, incredibly well documented, open source, and run by an awesome guy. It's an all around win-win.
  - stephenr 11y ago
    Rsync.net is even cheaper, has no requirement for a custom client, and arguably are more dependable because they're not just reselling S3 Edit: not to mention they offer actual support not just "contact the author" email link as a last resort.
    - GhotiFish 11y ago
      I contacted the author today. He responded to me in 30 seconds.
      - vacri 11y ago
        I assume he still has to sleep, at least on some days. :)
      - stephenr 11y ago
        Try in 18 hours. Can you call him when something fails? I'm not saying he isn't responsive I'm saying depending on a one-man-band who is responsible for the client software, server software and the underlying storage system (ie he is the owner of the s3 account) seems like a huge risk.
cperciva 11y ago
I suppose I should have known that this would end up at the top of Hacker News...
- JacobAldridge 11y ago
  It's the picodollars - tarsnap was the second business I fell in love with on HN (the late Kiko was #1) purely because of the awesome vibe I felt emanating from your enterprise (which I'm assuming is a reflection on you as well). Years later, you've also become a cause celebre for holding true to a clear business and lifestyle vision (again, perceived at distance), in spite of the recommendations and 'support' provided by Patrick and others, including myself. Keep being true, and I suspect the community will keep learning from you Colin.
  - cperciva 11y ago
    It's the picodollars Hey Thomas, are you listening here? In all seriousness, the picodollars do an excellent job of attracting exactly the sort of customers I want... and turning away the customers I don't want. They were originally part joke and part a way to avoid arguments with customers who don't understand that 1 GB < 1 GiB, but now it's way more than that. in spite of the recommendations and 'support' provided by Patrick and others Don't be too harsh on Patrick. His vision for Tarsnap is not my vision for Tarsnap, but he has helped me to orient myself: The projection of "business" onto the subspace "geek" doesn't look very much like "business", but it's not the same as "kid right out of university who has never had a real job" either, and that's what you would see if I hadn't had advice (from Patrick, Thomas, various YC people, and the rest of HN). Advice can be very valuable even if you don't follow it to the letter.
    - JacobAldridge 11y ago
      Don't get me wrong - I probably still agree with Patrick and Thomas. (Previous: https://news.ycombinator.com/item?id=7731268 https://news.ycombinator.com/item?id=7731268) [Edit: And I think the competing theories are an excellent lesson for that "kid right out of university who has never had a real job".]
Osiris 11y ago
For those that want to run a similar service using their own systems, I found that Attic [1] is a great open source backup tool that works in a very similar way, including deduplication and compression. I backup some VPS servers to my NAS at home using attic over an SSH tunnel. Incremental backups are quite small and it's easy to automate with a simple cron job. [1] https://attic-backup.org/ https://attic-backup.org/
- middleclick 11y ago
  How does this compare to Duplicity?
  - scott_karana 11y ago
    It uses git-style addressable blob storage, so you don't have to worry about deltas, because there aren't any. It's also got more efficient deduplication, because it doesn't use rsync's naïve algorithm. The downsides: it requires the agent to be remotely installed (a la rsync: no "dumb" backends), and supports less storage backends to boot. YMMV :-)
k1w1 11y ago
As an AWS user this type of thing gives me cause for concern: At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this immediately resulted in the service being I/O limited. A sudden doubling of latency can have dire consequences on any system. Knowing that such unexpected changes are possible makes it built trust in your environment, even if it is running fine today.
- cperciva 11y ago
  Indeed, I didn't know such a change was possible -- that EBS volume went for years with consistent low latency before it suddenly slowed down.
  - toomuchtodo 11y ago
    DevOps/Infrastructure engineer here! I see this happen frequently in AWS. Never expect either your instance networking latency or the latency of the underlying EBS storage layer to be consistent. If you absolutely need guaranteed IO performance, use an instance store or move to dedicated hardware. Them be the breaks of cloud computing. http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing http://en.wikipedia.org/wiki/Fallacies_of_distributed_comput...
  - jeffbarr 11y ago
    You could have contacted AWS support or emailed me. Either way, we would have investigated.
    - cperciva 11y ago
      It wasn't missing its guaranteed # of I/Os per second, so I figured the slowdown was just "one of those things" and not an out-of-spec issue. Happy to send you the volume ID if you think someone would want to investigate (and still has data from the start of April) though.
btmorex 11y ago
Why are you reinventing a scheduler when the OS (at least Linux) already provides a good one?
- cperciva 11y ago
  I'm talking about scheduling tasks within a single process.
  - btmorex 11y ago
    Threads
    - cperciva 11y ago
      Too much overhead. Also, concurrent systems are actively malicious.
      - btmorex 11y ago
        I don't believe that you have too many active connections for threading to work. Passive connections can be handled by a single or small number of threads. Modern Linux on modern hardware has no problem with many thousands of threads and the overhead is minimal in $$$ compared to the time you wasted debugging a scheduling problem. As for concurrent systems being harmful, you just have to design your program for threading in mind. Minimize shared state and be very careful when you can't.
        cperciva 11y ago
        One connection can have many outstanding requests.
        btmorex 11y ago
        I would redesign your protocol to be request/response based akin to http. Achieve performance by using multiple connections in the client. Simplicity > efficiency especially if you don't have the engineering resources of a company like Google. And I'm out. The reply rate limiting is infuriating.
Someone 11y ago
Good description, but I'm missing lesson learned #0: Do not wait too long before informing your users, even if only to tell them "we know about it and are working on it"

172 points by pndmnm 11y ago

This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems healthy or reassuring that incredibly smart people like Colin Percival suffer from similar challenges around fully understanding the scope of the problem and the solution. All that being said, I really respect the detailed response from a technical perspective as well as owning up to (and the decisions that went into) a spell of downgraded performance. Later edit because I don't want to spam the comments: I'd love some context (maybe from cperciva himself?) around the performance enhancement of integrating new Intel AESNI instructions. This is well beyond my depth and while Colin mentions that it didn't necessarily increase performance, I'm wondering if the hope is it would longterm? Or were there other benefits to such an integration?

Poiesis 11y ago
I came here to write this comment essentially. Very much of the time I feel, "If I knew what the problem[s] [was|were] it'd be solved by now!" That's not exactly true of course but of course diagnosis is a large part of the total solution. This type of an answer that Colin gave above does not exactly win friends and influence people in most situations where you're part of a team or hierarchy. Can anyone share what they've done to give better answers in these cases? I understand why people want the answers, but I don't have them to give right away particularly when it's Someone Else's system.
- cperciva 11y ago
  Just to be clear, when Tarsnap users wrote to me I told them everything I could. The "I think it will be fixed soon" delay in sending out an email to the lists affected only people who didn't notice or noticed but didn't ask about the issue.
- jballanc 11y ago
  One trick that I've learned (though I still have trouble routinely applying it myself) for these situations is: less is more. That is, as engineers we tend to want details. All the details. We want to know what happened, why it happened, how it's going to be fixed, and how long that will take. Because we want all that detail for ourselves, we hesitate to contact our customers/boss until we have all the details. Combine that with a desire to fix problems as they come up, and you end up with, "I never told you there was a problem because I was always one fix away from the solution." But most people are not engineers. They want to be acknowledged. They want to feel informed, even if they have less details than what you would like to provide for them. Sometimes, something as simple as, "We've noticed that there is an issue and are currently working on a fix," goes a long way. Also don't be afraid to pull out, "Users have been reporting issues with backup performance. We do not currently believe this represents a service failure, but we are working to return performance to normal levels." Your users trust you (otherwise they wouldn't pay you). If you "believe" something, they will too.
  - annnnd 11y ago
    This. There is also a saying on a similar note: "It's OK to disappoint, it's not OK to surprise."
    - psykovsky 11y ago
      What if the disappointment is a surprise?
cperciva 11y ago
I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed This ties in to the last lesson I mentioned at the bottom: 5. When performance drops, it's not always due to a single problem; sometimes there are multiple interacting bottlenecks. Every time I identified a problem, I was correct that it was a problem -- my failing was in not realizing that there were several things going on at once.
- mtsmith85 11y ago
  That totally jibes with what I found "reassuring" in a sense. That even very smart people sometimes get hit with inadvertent "multiple problems looking like a single issue" situations.
- jcrites 11y ago
  > Every time I identified a problem, I was correct that it was a problem -- my failing was in not realizing that there were several things going on at once. Very common! One thing that's been helpful for us is establishing predefined system performance thresholds that, if exceeded, initiate the chain of events that will lead to customer communication. "If X% of requests are failing, then we had better advertise that the system is degraded." Discussing and setting these thresholds in advance and the expectation that they'll result in communication helps drive the right outcome. It's not perfect, because one is always tempted to make a judgment call in the circumstance, which is vulnerable to the same effect, but it's a good start. Thanks for sharing!
- spydum 11y ago
  i tend to get to debug problems like this (usually in 3rd party code i dont know the internals of) pretty frequently.. my experience has been it tends to follow a curve..MOST of the time, the problem is simple and you can quickly dispatch it. the scary (or fun, depending on your perspective) part hits when you pass the first level, and there are still problems.. and you dont know if it's two or ten levels deeper. then you get into that crazy test/optimize cycle and crawl out two weeks later wondering when you last ate..
cperciva 11y ago
I'd love some context (maybe from cperciva himself?) around the performance enhancement of integrating new Intel AESNI instructions. I was using OpenSSL for that (which was using a software implementation). The code (you can see it in spiped) now detects the CPU feature and selects between AESNI or OpenSSL automatically. Given that the tarsnap server code was spending about 40% of its time running AES, it's a nontrivial CPU time saving. I should probably have been clearer in my writeup though -- using AESNI was never a "once I roll this out everything will be good" fix. Rather, it was a case of "I have this well-tested code available which will help a bit while I finish testing the real fixes".
- gonzo 11y ago
  One wonders why you aren't using a version of OpenSSL that has the AESNI bits already in it.
  - cperciva 11y ago
    I've learned that wondering about OpenSSL internals is detrimental to my sanity.
mryan 11y ago
This "it's almost fixed, I'll email the client soon" pattern is something I have personally struggled with a lot, and I agree it appears to be common with engineers. My workaround has been to make something else responsible for sending the email. In a team, this could be a manager setting a cut-off point after which communication must be made. When working on my own, I set an alarm for X minutes. When that alarm goes off I ignore the internal voice which says "just try one more thing, then send the email", and send an update to let the relevant people know my current progress, ETA to fix, and when they can expect the next update. I think this is similar to how GTD encourages us to use systems for storing to-do lists instead of trying to remember them - our fragile human brains are not always to be trusted.

patio11 11y ago

In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't trip our monitoring at the time because they all completed successfully. n.b. Our backups run outside of the hotspot times for Tarsnap, so we may have had less performance impact than many customers. I have an old habit of "Schedule all cron jobs to start predictably but at a random offset from the hour to avoid stampeding any previously undiscovered SPOFs." That's one of the Old Wizened Graybeard habits that I picked up from one of the senior engineers at my last real job, which I impart onto y'all for the same reason he imparted it onto me: it costs you nothing and will save you grief some day far in the future.

mtsmith85 11y ago
Hear hear on said Old Wizened Graybeard habit. The amount of pain inflicted from twenty jobs all starting up at :00 (or even :30, :45, etc.) when they could easily run at :04 or :17 can be huge. Anecdotally I once "lost" a sandbox server to a ton of developer sandbox jobs starting at :00 and not completing before the next batch started.
- protomyth 11y ago
  Funny part to that, was on a project with multiple teams with multiple crontabs. Each team took that advice to heart for some jobs. Sadly, we had too many Hitchhiker fans and :42 became a bit too common.
  - kijin 11y ago
    Use the following shell command to decide when to run cron jobs. echo $((RANDOM % 60)) It's not a CSPRNG, but good enough for this kind of load balancing!
    - 11y ago

deleted 11y ago

[deleted]

appsonify 11y ago

what the fu....Colin Percival used to be my cello teacher 12 years ago....and he is running tarsnap. My mind is blown.

cperciva 11y ago
Not me -- I'm a violinist, and I've never taught anyone violin either. Maybe you're thinking of my bother (Graham)? He was teaching cello around that time period I think.
- appsonify 11y ago
  Oh wow. Yes! Now I remember it was Graham. I was his student around then. Saw your photo on twitter and looked exactly like him! Hahaha. How is Graham? My mind is just completely blown right now.
  - cperciva 11y ago
    Graham is in Japan right now, but returning to Canada soon; he'll be joining me in the West Coast Symphony for our June concert. I don't feel that I should really be talking too much about my family in a public forum, but if you'd like to send me an email I can forward it to him.

ac29 11y ago

Sorry if this is offtopic, but can anybody explain the value proposition of tarsnap to me? It seems like a nice service and all, but the pricing is an order of magnitude more expensive than S3. If you are storing a few GB, this might not matter ("over half of Tarsnap users spend under $1 per month on storing their backups"), but if you have that little data, why not just dump it on a free Dropbox/Gdrive/etc account? For more data, why not just use one of the many compressed, deduplicated, encrypted, incremental backup systems (attic comes to mind, I'm sure there are others) then just sync to S3 at a tenth the cost?

segf4ult 11y ago
Because tarsnap is cheap, incredibly well documented, open source, and run by an awesome guy. It's an all around win-win.
- stephenr 11y ago
  Rsync.net is even cheaper, has no requirement for a custom client, and arguably are more dependable because they're not just reselling S3 Edit: not to mention they offer actual support not just "contact the author" email link as a last resort.
  - GhotiFish 11y ago
    I contacted the author today. He responded to me in 30 seconds.
    - vacri 11y ago
      I assume he still has to sleep, at least on some days. :)
    - stephenr 11y ago
      Try in 18 hours. Can you call him when something fails? I'm not saying he isn't responsive I'm saying depending on a one-man-band who is responsible for the client software, server software and the underlying storage system (ie he is the owner of the s3 account) seems like a huge risk.

cperciva 11y ago

I suppose I should have known that this would end up at the top of Hacker News...

JacobAldridge 11y ago
It's the picodollars - tarsnap was the second business I fell in love with on HN (the late Kiko was #1) purely because of the awesome vibe I felt emanating from your enterprise (which I'm assuming is a reflection on you as well). Years later, you've also become a cause celebre for holding true to a clear business and lifestyle vision (again, perceived at distance), in spite of the recommendations and 'support' provided by Patrick and others, including myself. Keep being true, and I suspect the community will keep learning from you Colin.
- cperciva 11y ago
  It's the picodollars Hey Thomas, are you listening here? In all seriousness, the picodollars do an excellent job of attracting exactly the sort of customers I want... and turning away the customers I don't want. They were originally part joke and part a way to avoid arguments with customers who don't understand that 1 GB < 1 GiB, but now it's way more than that. in spite of the recommendations and 'support' provided by Patrick and others Don't be too harsh on Patrick. His vision for Tarsnap is not my vision for Tarsnap, but he has helped me to orient myself: The projection of "business" onto the subspace "geek" doesn't look very much like "business", but it's not the same as "kid right out of university who has never had a real job" either, and that's what you would see if I hadn't had advice (from Patrick, Thomas, various YC people, and the rest of HN). Advice can be very valuable even if you don't follow it to the letter.
  - JacobAldridge 11y ago
    Don't get me wrong - I probably still agree with Patrick and Thomas. (Previous: https://news.ycombinator.com/item?id=7731268 https://news.ycombinator.com/item?id=7731268) [Edit: And I think the competing theories are an excellent lesson for that "kid right out of university who has never had a real job".]

Osiris 11y ago

For those that want to run a similar service using their own systems, I found that Attic [1] is a great open source backup tool that works in a very similar way, including deduplication and compression. I backup some VPS servers to my NAS at home using attic over an SSH tunnel. Incremental backups are quite small and it's easy to automate with a simple cron job. [1] https://attic-backup.org/ https://attic-backup.org/

middleclick 11y ago
How does this compare to Duplicity?
- scott_karana 11y ago
  It uses git-style addressable blob storage, so you don't have to worry about deltas, because there aren't any. It's also got more efficient deduplication, because it doesn't use rsync's naïve algorithm. The downsides: it requires the agent to be remotely installed (a la rsync: no "dumb" backends), and supports less storage backends to boot. YMMV :-)

k1w1 11y ago

As an AWS user this type of thing gives me cause for concern: At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this immediately resulted in the service being I/O limited. A sudden doubling of latency can have dire consequences on any system. Knowing that such unexpected changes are possible makes it built trust in your environment, even if it is running fine today.

cperciva 11y ago
Indeed, I didn't know such a change was possible -- that EBS volume went for years with consistent low latency before it suddenly slowed down.
- toomuchtodo 11y ago
  DevOps/Infrastructure engineer here! I see this happen frequently in AWS. Never expect either your instance networking latency or the latency of the underlying EBS storage layer to be consistent. If you absolutely need guaranteed IO performance, use an instance store or move to dedicated hardware. Them be the breaks of cloud computing. http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing http://en.wikipedia.org/wiki/Fallacies_of_distributed_comput...
- jeffbarr 11y ago
  You could have contacted AWS support or emailed me. Either way, we would have investigated.
  - cperciva 11y ago
    It wasn't missing its guaranteed # of I/Os per second, so I figured the slowdown was just "one of those things" and not an out-of-spec issue. Happy to send you the volume ID if you think someone would want to investigate (and still has data from the start of April) though.

btmorex 11y ago

Why are you reinventing a scheduler when the OS (at least Linux) already provides a good one?

cperciva 11y ago
I'm talking about scheduling tasks within a single process.
- btmorex 11y ago
  Threads
  - cperciva 11y ago
    Too much overhead. Also, concurrent systems are actively malicious.
    - btmorex 11y ago
      I don't believe that you have too many active connections for threading to work. Passive connections can be handled by a single or small number of threads. Modern Linux on modern hardware has no problem with many thousands of threads and the overhead is minimal in $$$ compared to the time you wasted debugging a scheduling problem. As for concurrent systems being harmful, you just have to design your program for threading in mind. Minimize shared state and be very careful when you can't.
      - cperciva 11y ago
        One connection can have many outstanding requests.
        btmorex 11y ago
        I would redesign your protocol to be request/response based akin to http. Achieve performance by using multiple connections in the client. Simplicity > efficiency especially if you don't have the engineering resources of a company like Google. And I'm out. The reply rate limiting is infuriating.

Someone 11y ago

Good description, but I'm missing lesson learned #0: Do not wait too long before informing your users, even if only to tell them "we know about it and are working on it"