6 ms·

Ask HN: My VPS got hacked and now I'm facing a massive bill. What can I do?

• theonemind 12y ago
  I work for a company that provides VPSes. In a situation like this, they can see the usage is aberrant and they can see it's not normal based on past bills. They'd likely offer a large credit if you say you didn't intend to do this, and it doesn't look like a fraudulent account. That being said, they themselves probably have bandwidth costs, and are not at all likely to forget all of the charge, perhaps half at best.
• deleted 12y ago
  [deleted]
• Jare 12y ago
  Depends on your provider. Amazon AWS is known to have waived such bills in the past, see for example http://readwrite.com/2014/04/15/amazon-web-services-hack-bitcoin-miners-github http://readwrite.com/2014/04/15/amazon-web-services-hack-bit...
• patio11 12y ago
  I would begin by contacting your VPS provider, explaining the circumstances which caused the bill, and asking "What are our options?"
• ColinCera 12y ago
  Have you talked to your VPS provider? They should be able to cut you a break; after all, that 40TB of traffic cost them only a small fraction of what they're charging you, so if they're reasonable you should at least be able to get them to reduce the charges to their actual cost. You might also offer to suggest writing up a post mortem for them, that they can provide to their customers as a lesson/tutorial on how to protect a VPS. Finally, you can suggest that they might want to implement (and perhaps help them implement it) some kind of warning system, i.e., if a VPS suddenly begins using exorbitant amounts of bandwidth, and far more bandwidth than it ever has before, they really should email/text the owner an alert within 24 hours — not let it go on for 6 weeks. I'm surprised that they don't cap/throttle the bandwidth once you go over your plan's limit, to go along with sending you alerts. It borders on negligence on their part that they don't already have such a system in place.
  • MangoDiesel 12y ago
    In my opinion, it is negligence to an extent that OP should not have to pay for this, and he should find a new VPS provider.
    • Khaine 12y ago
      Why? He failed to secure his server. Why is that the fault of the VPS Provider?
• minopret 12y ago
  I can understand how that could happen and what a problem it would be. I had an experience with a telephone bill myself, but the story is not going to help you. I would suppose your first and best resort is to consult your lawyer, advocate, solicitor, barrister, Anwalt. I wonder what your relevant legal jurisdiction is. I wonder whether it would help if you can account for your own whereabouts and your own usage of endpoint data services. I wonder if your method of payment to your VPS provider is mediated by a financial service that can help you dispute the bill. I am not a lawyer.
• ishener 12y ago
  I feel really sorry for you situation. I first suggest talking to the hosting provider and explain what happened. Any decent service will give some discount in this case. Unfortunately, I can't think of anything else. I wish it was realistic to tell you to go to the police. Also, if you would give your email, I would definitely consider sending a donation through paypal... Hopefully other readers here will do the same.
  • patio11 12y ago
    There is one great reason to go to the police: it establishes a paper trail documenting that a crime was committed. It isn't necessary that the police catch the bad guys for that paper trail to be advantageous. (Examples: police reports make CC disputes and legal declarations much easier and more likely to be given weight as other than self-serving explanations of a deadbeat. It may also trigger insurance policies either for you or for the VPS company.)
    • ishener 12y ago
      ok, so he should go to the police but that has nothing to do with helping HIM...
      • patio11 12y ago
        Sorry for being pretty brief earlier, as I was on a cell phone. Here's how a police report can help him, in a few ways: 1) In dealing with the company that is billing you, a police report sends a strong, costly signal that "This is not a routine commercial transaction! A crime has been committed!" which will throw an exception in the billing process that will get caught elsewhere in the company. The accounts receivable department is scored on collecting revenue. That is, to a first approximation, the only thing they care about. The legal department, anti-abuse department, etc etc, is not scored on revenue impact. You're much more likely to have a productive conversation with them that involves waiving the $4k than you will with Accounts Receivable. AR's calculation goes something like "We'd prefer getting $4k over getting $0." The rest of the company thinks "We'd prefer writing off an uncollected invoice -- costs us $0 -- versus spending hundreds of dollars an hour dealing with the legal process for an unbounded amount of time." 2) Assuming that European credit card providers work on the same heuristics as US ones, "I filed a police report" means their response to your chargeback will virtually invariably be "Wait, an actual criminal case? Eff that, sorry merchant, we do not get paid to litigate on your behalf. Best of luck figuring this out after we give the customer their money back." (Conversely they use police reports as a bar for treating claims of fraud seriously, for example, in the case of "family fraud." If you ever say, for example, "My wife bought that but I didn't let her?" the very next question will be "Have you filed a police report against her for stealing from you?" and if you haven't the very next thing they'll say will be "Well, work it out with her then -- the charge stands. Thxmuchbye!") 3) "Theft" is such a wonderfully useful word for dealing with insurance companies. (HNers might say "No theft happened! It was actually..." No, guys, really, theft happened.) It may be covered under your homeowner's insurance -- depends on the particulars of your policy -- and would almost certainly be covered if you had business insurance. It also may be covered under the VPS provider's insurance, in which case they may decide that rather than trying to ring water from a stone (you) that filing a claim is the quicker way to get paid. (I rather doubt, as a businessman with an insurance policy, that they'd jeopardize their future insurability over a $4k claim, because that is tiddliwinks, but risk management is exactly why that policy exists.)
• jnardiello 12y ago
  I assume you are in europe. I'd suggest simply talking with your provider, explaining the issue and asking them to investigate. I honestly expect them to cooperate and be understanding. If they insist for you to pay: simply don't. State the truth: You can't afford it. Tell them the only way they will see this money is by taking legal action against you and even in that case you won't be able to comply - as you don't have the money. Hope it helps :(
• Blahah 12y ago
  1. Report the incident to the police. Right now. 2. Report it to the VPS provider. Explain that you've reported it to the police. Ask for their cooperation in investigating the problem. You do not have to pay. If they try to force you to pay, depending on your country, you'll probably end up in small claims court where you'll find judges are very reasonable people who usually side with the little guy. (IANAL)
  • minopret 12y ago
    I wonder what experiences we have with police in similar situations in recent years. In the USA many years ago I brought a telephone bill dispute to police. They explained, not without kindness and patience, that the legal question involved intent. Consequently it wasn't a matter where they could easily provide assistance.
    • hyperpape 12y ago
      What was the situation? In the OP's case, they're alleging a very definite crime of illegally accessing a VPN to do nasty things with it. I think at least taking a report is going to be standard behavior, though I wouldn't be surprised if they just let it "sit on the shelf". If it's just a garden variety dispute with a company, then it's more likely to be a) a civil matter, and b) subject to interpretation.
      • minopret 12y ago
        Well, that's the point. The specifics make a difference. In my example, it appeared that a roommate unknowingly permitted a friend of theirs that I never met then or since to run up big charges at billable services on the apartment telephone line that was in my name. (I understand what my mistake was and because it was so long ago it's not a sore issue.)
  • monstermonster 12y ago
    Yes. This happened to a company I worked for in the UK in the last 1990s. We had a half rack full of NT4 machines and someone got into our kit and used it to run a pr0n FTP. They took us to court to pay up and the magistrate said we had no bill to pay and that it was a waste of court time as it wasn't intentional. You're right about reporting a crime though even if the police don't take it seriously. A crime ref number goes a long way on its own. Edit: we had to move our kit sharpish though as the company exercised their right to throw it on the street within 24 hours.
• BukhariH 12y ago
  I've been in the exact same situation with AWS ( http://cl.ly/SHOu http://cl.ly/SHOu ). It was a nerve recking couple of days but I contacted AWS support and they were extremely good. They helped me secure my machine and then cancelled the 1.4K payment they were going to take from my account. In all the whole process took 2.5 weeks and I only had to pay $15 for the I/O requests. The best thing I can recommend is to talk to your host and tell them honestly you can't pay that much and you weren't the cause of the charges either.
  • malditojavi 12y ago
    Same here, not as big as 4k, but only $200 more. AWS took care in an awesome way.
  • e40 12y ago
    I've seen AWS credit charges that people intended to make but didn't want. We have some public AMIs that include charges for our product. Twice AWS support contact us to ask if we would credit someone back on the order of $500 out of our pocket because their customer didn't realize running an AMI would incur charges. Yeah, I was pretty surprised by this, too. It seems AWS gives people one freebie, though.
    • guan 12y ago
      Did you grant the credit? Why or why not?
      • e40 12y ago
        We did because AWS did and we didn't want to look bad.
  • rmc 12y ago
    In aws, you can set up billing alerts, so they will email you if you go over X per month. It's a good idea to set that up, so at least you'll be alerted as soon as possible if you get hacked.
• freshflowers 12y ago
  Just in addition to some other helpful comments: based on posting I assume that your are Dutch or Belgian, located in Europe and are buying this VPS as a private consumer, not a company. Which means your case is probably covered by consumer protection rules when it comes to informing you about data usage, and I seriously doubt a VPS provider has covered their ass as well as mobile providers tend to do.
• general_failure 12y ago
  Do you know how you got hacked?
• zhovner 12y ago
  To prevent such incidents Linode have alerts of traffic/cpu/disk thresholds. For example you can configure notification if your bandwidth utilization more than N Mbit/s in duration more than N minutes. Very useful for DDoS prevention.
  • emeraldd 12y ago
    Make sure you tune those alerts appropriately and know when your systems should be doing things that will trip them. We've had a couple of boxes that regularly trip cpu warnings during normal operation, but only inside specific time windows. Knowing what's not normal is vitally important with this stuff.
  • bluedino 12y ago
    And look at your dashboards once in a while. I'd find it unusual if I saw my toy VPS cranking out 100mbs for a week straight!
• dangoldin 12y ago
  I had something similar happen with AWS but the bill wasn't as high since they ended up flagging my box as spam-producing and shut off all outbound traffic. I'd just ask them and see if they can remove the charges, it worked in my case.
• onestone 12y ago
  Stop using providers which charge a ridiculous price for bandwidth (like AWS). There are many excellent alternatives where a TB costs only a few dollars/euros.
  • njsubedi 12y ago
    like?
    • notok22 12y ago
      Hetzner has the first 20 TB free and then charges 2 euros per TB.
      • chhantyal 12y ago
        This.
      • JosephRedfern 12y ago
        Beware: https://news.ycombinator.com/item?id=6577465 https://news.ycombinator.com/item?id=6577465
        • Nyr 12y ago
          Any sane provider with no DDoS protection will nullroute you on incoming DDoS, that's not Hetzner specific. If you expect to get DDoSed, buy protection or go with OVH.
          • vacri 12y ago
            The problem wasn't the nullrouting, it was 'contact support to re-enable... and support isn't open until Monday'.
    • onestone 12y ago
      Linode, DigitalOcean, OVH, Hetzner, and so on...
    • 12y ago
• applecore 12y ago
  PSA: Set up billing alerts! You should always have a notification sent to you when your monthly bill exceeds one or more dollar amounts. For example, if you're using AWS, Amazon CloudWatch lets you set an alarm on a billing metric to notify you automatically.
• gregcmartin 12y ago
  Make sure Elastic Search is not accessible from a public IP address (this is what likely got you in the mess to begin)
• patio11 12y ago
  Incidentally, since many HNers probably come at this from a mental model of "Anything which appears on an invoice is non-negotiable and simply must be paid": a B2B service provider which collects payment after services are rendered is knowingly taking on credit risk and has already priced non-collectability of some accounts into their services. You may be overestimating how much drama is required for someone at their company to say "Wow, really? OK, sorry about that. I'll write it off." This is one of many, many, many reasons why we don't generally do cost-based pricing and, when we do do cost-based pricing, the markup is absolutely phenomenal. It has to include risk premiums. As long as it do include risk premiums, you don't have to sweat the small stuff like e.g. an uncollectable $4k invoice. (n.b. Small stuff! $4k hiccups are utterly routine events and largely dealt with by processes rather than by treating them as sudden emergencies, even if they feel like that to natural humans.)
  • 13 12y ago
    I have had this experience. In my case the bill was altered to $0 before I'd even got to the point in the conversation where I intended to ask for some relief. It most certainly doesn't cost them that much, and it's within their interests to keep customers coming back for repeat payments than soured by being forced to pay an exorbitant fee for something which was obviously not their fault.
    • justizin 12y ago
      Yeah, the $4k is kind of a manufactured figure, bandwidth expenses are particularly inflated, but it makes sense at smaller scale - if you have a $15/mo plan with 2TB of transfer and you go over 1TB, a few bucks penalty doesn't sound outrageous, and their cost is impacted by higher network management costs. But, if you have a huge spike that wasn't really your fault, it doesn't cost them any more to write that off than it does the bandwidth that is consumed by a DDoS attack that is mitigated by a firewall. When I worked at Rackspace, we hosted a very popular flash cartoon for one month after yahoo kicked them off the $5 hosting plan for pushing god knows how much bandwidth. They basically saturated a gigabit network port from a single server, and we sent them a bill for like a gajillion dollars. They went to another hosting company, of course, and I think got the bill down to something they managed to pay off. Someone obviously had porsche-eyes, I thought it was kind of a shitty thing to do to the guys, who came to the office to make the voices of their characters for us and stuff.
• bhaisaab 12y ago
  If you want a VPS with unmetered network you should consider http://scale.ninja http://scale.ninja
• thewhk 12y ago
  I work for a VPS provider in the US. These situations are common and we usually just issue a credit and give a reminder to the customer to please secure their server That brings me to my point. How did the hack occur? When you get a VPS you are fully responsible for what goes on in there. It is your responsibility to secure it and keep it updated. It's not the provider's fault you did not apply the latest security updates. It's not the provider's fault your Java application was using outdated and vulnerable libraries nor is it their fault you didn't set a CAPTCHA in front of your submission forms. Either hire a competent sysadmin if you can't take care of that yourself or find a provider that offeres managed hosting instead of a VPS, as that's what you'd most likely need. There are some cases where it's the provider's fault such as the Linode BitCoin hack a few years back but mostly it's just poor server maintenance
  • waxjar 12y ago
    People that rent a $15/year VPS use it to run an IRC bouncer or a small web log, something you don't need to know a whole lot of sysadmin stuff for. They just need a machine that's always on. It's hardly worth hiring a sysadmin for (I find that suggestion laughable, to be frank). Managed hosting doesn't allow you to do much else besides hosting a website in PHP, which is not enough for plenty of use-cases, including OPs.
    • lucb1e 12y ago
      > $15/year VPS Do tell, where do I get one of those? Cheapest I know of is $60 ($5 a month).
      • xrjn 12y ago
        http://serverbear.com/compare/vps http://serverbear.com/compare/vps is a great place to find these VPSs, they list some of the smaller services.
      • briandh 12y ago
        Check http://lowendbox.com/ http://lowendbox.com/ , though YMMV.
• tdicola 12y ago
  Anyone have tips on how to secure their Linux VPS? I just set one up and disabled SSH password login, locked down all the ports with iptables (using ufw), and enabled fail2ban. Anything else I should install or configure to make myself a little more secure? Was considering tripwire but I dunno how much a headache it would be with false positives as I change things on the server.
  • MayIHaveAnother 12y ago
    A very common attack vector is through installed web applications. Especially if you run wordpress with a lot of plugins installed, be sure to enable correct read/write settings for /var/www, and update your application frequently. Malicious entities runs 24/7 scans towards indexed URL's attempting to exploit various vulnerabilities, and many of the vulnerabilities allows remote code execution, upload of php files etc. This can be used to upload malicious code, simple php-webshells, and then your VPS is suddenly a part of a DDoS/Scanning network. Exploited Wordpress sites are a problem, Zeus/Zbot-Trojan is often seen downloading updates/configs from these, and they are also often used to redirect users to Exploit Kits.
    • porker 12y ago
      I came across these two tools recently that seem interesting: http://www.rfxn.com/projects/linux-malware-detect/ http://www.rfxn.com/projects/linux-malware-detect/ https://github.com/emposha/PHP-Shell-Detector https://github.com/emposha/PHP-Shell-Detector Not installed either yet (LMD could really use some .deb packages) but could be a useful alternative to Tripwire
  • Ded7xSEoPKYNsDd 12y ago
    On my personal machines, I also setup a cronjob to automatically install updates. There's a small risk of breakage (I had one in 5 years), but I prefer that over a bill like op's.
• logn 12y ago
  In addition to the other comments, make it absolutely clear to them (with proof if needed) that you're a student.
• zack19 12y ago
  cut your credit card report it as stolen tell the host that it wasn't u :p
• matthewarkin 12y ago
  I had the same thing happen to me. I wrote about it on my blog http://mattarkin.com/protect-your-azure-linux-vm-aka-how-to-avoid-a-1500-charge/ http://mattarkin.com/protect-your-azure-linux-vm-aka-how-to-.... Basically I complained to Microsoft, they said they'd waive the charge but since it was for a linux vm they said they couldn't cover it. Then I complained to American Express claiming it was an unauthorized and fraudulent charge. Amex sent the dispute to Microsoft and they never responded so I wont the chargeback.
• decisiveness 12y ago
  I seem to be missing something. You knew it was happening when you got the first bill, but let it continue for another half month before shutting it down?
• joshmn 12y ago
  Post on WebHostingTalk.com - just do it. You'll get attention from the host, other hosts who will sympathize, and you'll see that they'll just write it off. Post the link when you do and I'll be sure to comment on it (I'm somewhat very-active at WHT)