13 ms·

That’s a really fair and reasonable response. Not sure what else people really expect here. > The template used for response in account denial will be removed

That’s a really fair and reasonable response. Not sure what else people really expect here. > The template used for response in account denial will be removed entirely. If account access is denied during an appeal, which often is the case as most appeals are true bad actors, the agent must create a reasoned response. Glad this is seen as an issue and corrected. IMO, this probably would have made this whole thing never escalate if a better response was previously in place for everyone. Accidents, shotty support, whatever — all expected these days unless you have big cash money agreements in place. But to kill an account of a responsive person with a gigantic middle finger email without reasoning was a pretty dumb process in place. You can see the email on the Twitter thread somewhere. Glad it’s fixed! Still a DO fan here Edit: TALKING ABOUT THIS: https://pbs.twimg.com/media/D76ocofXoAY_xB5.png https://pbs.twimg.com/media/D76ocofXoAY_xB5.png

Angostura 7y ago
> shotty support "shoddy", for what it's worth.
- scotchio 7y ago
  Woop. TIL - Thanks!
solidasparagus 7y ago
I can't really fault their postmortem or their response on HN. The corrections are all good, but the very fact that these things need to be corrected (automatically locking the entire account when there is a compute spike, having such a casual review process before permanently denying access to an account, not having 24/7 support after locking an account, etc.) makes you question their overall maturity as a B2B infrastructure provider.
- TheOperator 7y ago
  Sure it's better to never make a mistake but so long as they don't make a habit of things like this I'm not going to think anything of it until I see more cracks in the wall. A screw up is inevitable. A mature response is not. So the fact they gave mature response goes a long way. Although it's unfortunate that social media seems to be their emergency support channel...
  - noselasd 7y ago
    > Sure it's better to never make a mistake but so long as they don't make a habit of things like this This is the thing - the customer that got locked out managed to get attention on HN, Reddit and other media - this seems to have prompted action from Digital Ocean. How many have silently fallen victim before this ? We don't really know if this is a habit or not - we only know this one customer was corrected.
    - ensignavenger 7y ago
      Based on this post, Digital Ocean is taking specific measures company wide to prevent similiar issues from affecting any customer in the future. So they did no just correct the situation for this one customer.
wwweston 7y ago
I agree it's a generally good response. There are a few more things I'd like to see more clearly addressed: * While the removal of the account termination template is good, in conjunction with additional hiring to support more attention to any individual ticket, I can't tell by whose standards the "reasoned response" is gauged, or if the response is reviewable at all. I did note that they now want two human reviewers, but that's distinct from specifying a process in which a reasoned response is articulated and reviewed. * More importantly, if the reasoned response doesn't pass muster with the customer, what's their resort? Still Twitter-shaming? I suppose that's legit if they'd rather their mistakes were public like this. * The question of whether an account-wide lockout w/ no data retrieval is a necessary/proper consequence for those flagged for CPU abuse needs addressing -- ideally they should have a different policy that allows for data egress (with bandwidth fees, if necessary), but if not, a rationale and clear policy might be acceptable.
- ensignavenger 7y ago
  Back in the days before Twitter, folks wrote to the CEO or other senior executives as a last resort. Might still be effective in some cases.
donmcronald 7y ago
> declined to activate it Except they were declining to unlock it, right? I’m always shocked to see support that’s so pitiful they don’t even bother to have a correctly worded template for a common event. The real problem is support reps that aren’t trained properly and don’t even care enough to apply a bit of common sense. Getting rid of a response template doesn’t automatically make the support reps care enough to apply common sense. How about a “don’t fuck me” support tier where I can pay a one time $100-$250 fee for the sole purpose of getting a phone call before my account gets banned?
- deleted 7y ago
  [deleted]
- close04 7y ago
  The real problem is most definitely not the support rep. They don’t really go off book. This is the process as designed and approved by higher management, not by a low pay first level support (unless you assume they have some top level engineer doing this stuff). And going off process could make it better... yay, self pat on the back. But it could make it worse in which case I see unemployment in the support rep’s future. So they won’t go that way very often. Anyone who ever had such a low positioned job knows how it works. At that level your only freedom is to do what you’re told and follow company process. No, this is the fault of the manager who asked for this process and their manager who approved it. Management isn’t just about picking up a higher paycheck, it’s also to take the accountability for the decisions made under your watch.
sergiosgc 7y ago
> That’s a really fair and reasonable response. Not sure what else people really expect here. The root cause of suspension is incomprehensible to me. They were suspended because they launched a set of instances and these were using 100% CPU. How is that unreasonable and cause for suspension? I'm not a Digital Ocean customer, but if I were, I'd expect to be able to use the resources I bought without risk of being suspended. This is the root cause. It was compounded by incompetent customer support, but I really do not understand the suspension cause. The response tackles all secondary factors, but does not talk about the root cause. I'd expect it to.
- numlock86 7y ago
  Having your instances run at 100% CPU pretty much raises a red flag at any cloud provider. Depending on your plan it either gets shut off (like in this case) or you get a notice about "suspicious" behavior and a bit of time to fix the "issue".
  - arianvanp 7y ago
    That sounds odd to me. Especially given that digitalocean is the default dynamic provider for Gitlab CI builds which _will_ run droplets at 100% CPU.
    - robteix 7y ago
      From what I understand, they’re not saying you’re not allowed to use 100% for (what their user agreements define as) legitimate uses. They’re saying several droplets suddenly created and immediately going to 100% flags them as suspicious activity for human review. Looks like after such review, they would flag them as legitimate and all would be fine, 100% CPU or not. They’ve botched that second step though.
      - numpad0 7y ago
        Not sure what the materialization of that suspicion might look like -- competitors trying to crush DO's business? mass account creation or mass fraudulent logins? "mining crypto"? What I could come up felt quote-unquote legit grounds for a timed suspension but only instinctively so.
EasyTiger_ 7y ago
How tremendously forgiving.
crankylinuxuser 7y ago
> That’s a really fair and reasonable response. Not sure what else people really expect here. If you nuke VMs, under no circumstances do you also nuke access to data, backups, etc. Because if it wasn't for "social escalation" (aka: mob justice via HN and Twitter), this 2 person company would have lost everything. If you terminate a customer for $reasons, the data still belongs to the user, and not the company. And the company should still be legally required to provide the data on a reasonable timescale, like FTP access for 7 days.
- corobo 7y ago
  While you're swinging that legal word around, have an armchair lawyer skim DO's Terms of Service. > 9.1 Subscriber is solely responsible for the preservation of Subscriber's data which Subscriber saves onto its virtual server (the "Data"). EVEN WITH RESPECT TO DATA AS TO WHICH SUBSCRIBER CONTRACTS FOR BACKUP SERVICES PROVIDED BY DIGITALOCEAN, TO THE EXTENT PERMITTED BY APPLICABLE LAW, DIGITALOCEAN SHALL HAVE NO RESPONSIBILITY TO PRESERVE DATA. DIGITALOCEAN SHALL HAVE NO LIABILITY FOR ANY DATA THAT MAY BE LOST, OR UNRECOVERABLE, BY REASON OF SUBSCRIBER'S FAILURE TO BACKUP ITS DATA OR FOR ANY OTHER REASON. Summary: Do offsite backups n'all you dinguses
  - megous 7y ago
    You're right. It's good to have some expectations of the company, but customers really need to take the TOS seriously.
  - crankylinuxuser 7y ago
    It's yet again that the ToS is hidden crap that goes against the direct things, like "Backups". Sure the ToS needs legalese crap for the lawyers, but a plain version also needs to be made. I'm certainly no lawyer, and nor are most people.
    - corobo 7y ago
      It’s not really hidden and they do have an easy to read non-lawyer summary underneath the part I quoted > In other words, we trust that you’ll be responsible and back up your own data. Things happen!

Angostura 7y ago

> shotty support "shoddy", for what it's worth.

scotchio 7y ago
Woop. TIL - Thanks!

solidasparagus 7y ago

I can't really fault their postmortem or their response on HN. The corrections are all good, but the very fact that these things need to be corrected (automatically locking the entire account when there is a compute spike, having such a casual review process before permanently denying access to an account, not having 24/7 support after locking an account, etc.) makes you question their overall maturity as a B2B infrastructure provider.

TheOperator 7y ago
Sure it's better to never make a mistake but so long as they don't make a habit of things like this I'm not going to think anything of it until I see more cracks in the wall. A screw up is inevitable. A mature response is not. So the fact they gave mature response goes a long way. Although it's unfortunate that social media seems to be their emergency support channel...
- noselasd 7y ago
  > Sure it's better to never make a mistake but so long as they don't make a habit of things like this This is the thing - the customer that got locked out managed to get attention on HN, Reddit and other media - this seems to have prompted action from Digital Ocean. How many have silently fallen victim before this ? We don't really know if this is a habit or not - we only know this one customer was corrected.
  - ensignavenger 7y ago
    Based on this post, Digital Ocean is taking specific measures company wide to prevent similiar issues from affecting any customer in the future. So they did no just correct the situation for this one customer.

wwweston 7y ago

I agree it's a generally good response. There are a few more things I'd like to see more clearly addressed: * While the removal of the account termination template is good, in conjunction with additional hiring to support more attention to any individual ticket, I can't tell by whose standards the "reasoned response" is gauged, or if the response is reviewable at all. I did note that they now want two human reviewers, but that's distinct from specifying a process in which a reasoned response is articulated and reviewed. * More importantly, if the reasoned response doesn't pass muster with the customer, what's their resort? Still Twitter-shaming? I suppose that's legit if they'd rather their mistakes were public like this. * The question of whether an account-wide lockout w/ no data retrieval is a necessary/proper consequence for those flagged for CPU abuse needs addressing -- ideally they should have a different policy that allows for data egress (with bandwidth fees, if necessary), but if not, a rationale and clear policy might be acceptable.

ensignavenger 7y ago
Back in the days before Twitter, folks wrote to the CEO or other senior executives as a last resort. Might still be effective in some cases.

donmcronald 7y ago

> declined to activate it Except they were declining to unlock it, right? I’m always shocked to see support that’s so pitiful they don’t even bother to have a correctly worded template for a common event. The real problem is support reps that aren’t trained properly and don’t even care enough to apply a bit of common sense. Getting rid of a response template doesn’t automatically make the support reps care enough to apply common sense. How about a “don’t fuck me” support tier where I can pay a one time $100-$250 fee for the sole purpose of getting a phone call before my account gets banned?

deleted 7y ago
[deleted]
close04 7y ago
The real problem is most definitely not the support rep. They don’t really go off book. This is the process as designed and approved by higher management, not by a low pay first level support (unless you assume they have some top level engineer doing this stuff). And going off process could make it better... yay, self pat on the back. But it could make it worse in which case I see unemployment in the support rep’s future. So they won’t go that way very often. Anyone who ever had such a low positioned job knows how it works. At that level your only freedom is to do what you’re told and follow company process. No, this is the fault of the manager who asked for this process and their manager who approved it. Management isn’t just about picking up a higher paycheck, it’s also to take the accountability for the decisions made under your watch.

sergiosgc 7y ago

> That’s a really fair and reasonable response. Not sure what else people really expect here. The root cause of suspension is incomprehensible to me. They were suspended because they launched a set of instances and these were using 100% CPU. How is that unreasonable and cause for suspension? I'm not a Digital Ocean customer, but if I were, I'd expect to be able to use the resources I bought without risk of being suspended. This is the root cause. It was compounded by incompetent customer support, but I really do not understand the suspension cause. The response tackles all secondary factors, but does not talk about the root cause. I'd expect it to.

numlock86 7y ago
Having your instances run at 100% CPU pretty much raises a red flag at any cloud provider. Depending on your plan it either gets shut off (like in this case) or you get a notice about "suspicious" behavior and a bit of time to fix the "issue".
- arianvanp 7y ago
  That sounds odd to me. Especially given that digitalocean is the default dynamic provider for Gitlab CI builds which _will_ run droplets at 100% CPU.
  - robteix 7y ago
    From what I understand, they’re not saying you’re not allowed to use 100% for (what their user agreements define as) legitimate uses. They’re saying several droplets suddenly created and immediately going to 100% flags them as suspicious activity for human review. Looks like after such review, they would flag them as legitimate and all would be fine, 100% CPU or not. They’ve botched that second step though.
    - numpad0 7y ago
      Not sure what the materialization of that suspicion might look like -- competitors trying to crush DO's business? mass account creation or mass fraudulent logins? "mining crypto"? What I could come up felt quote-unquote legit grounds for a timed suspension but only instinctively so.

EasyTiger_ 7y ago

How tremendously forgiving.

crankylinuxuser 7y ago

> That’s a really fair and reasonable response. Not sure what else people really expect here. If you nuke VMs, under no circumstances do you also nuke access to data, backups, etc. Because if it wasn't for "social escalation" (aka: mob justice via HN and Twitter), this 2 person company would have lost everything. If you terminate a customer for $reasons, the data still belongs to the user, and not the company. And the company should still be legally required to provide the data on a reasonable timescale, like FTP access for 7 days.

corobo 7y ago
While you're swinging that legal word around, have an armchair lawyer skim DO's Terms of Service. > 9.1 Subscriber is solely responsible for the preservation of Subscriber's data which Subscriber saves onto its virtual server (the "Data"). EVEN WITH RESPECT TO DATA AS TO WHICH SUBSCRIBER CONTRACTS FOR BACKUP SERVICES PROVIDED BY DIGITALOCEAN, TO THE EXTENT PERMITTED BY APPLICABLE LAW, DIGITALOCEAN SHALL HAVE NO RESPONSIBILITY TO PRESERVE DATA. DIGITALOCEAN SHALL HAVE NO LIABILITY FOR ANY DATA THAT MAY BE LOST, OR UNRECOVERABLE, BY REASON OF SUBSCRIBER'S FAILURE TO BACKUP ITS DATA OR FOR ANY OTHER REASON. Summary: Do offsite backups n'all you dinguses
- megous 7y ago
  You're right. It's good to have some expectations of the company, but customers really need to take the TOS seriously.
- crankylinuxuser 7y ago
  It's yet again that the ToS is hidden crap that goes against the direct things, like "Backups". Sure the ToS needs legalese crap for the lawyers, but a plain version also needs to be made. I'm certainly no lawyer, and nor are most people.
  - corobo 7y ago
    It’s not really hidden and they do have an easy to read non-lawyer summary underneath the part I quoted > In other words, we trust that you’ll be responsible and back up your own data. Things happen!