15 ms·
Verelox Wiped by Ex-Admin
- ceejayoz 9y agoYooooowch. They appear to be VPS and dedicated host.
- rexicus 9y agoThere goes the 99.95% uptime guarantee.
- oneplane 9y agoIn theory that still gives you a lot of downtime in a 365-day span :p Then again, once you are down for more than 4 hours (as they are), that uptime starts ticking away rather fast.
- wujonesj2 9y agoI believe a 99.95% uptime guarantee indicates that you will only have 4.38h of downtime _per year_. So, yeah, they've blown their allowance for the year and now need to execute perfectly.
- preinheimer 9y agoWayback Machine link if you want to know who they were: https://web.archive.org/web/20170603212121/https://verelox.com/ https://web.archive.org/web/20170603212121/https://verelox.c...
- michaelbuckbee 9y agoWow, they were a VPS hosting provider. Lots of unhappy customers (and this likely qualifies as a data breach for all of them as well).
- __jal 9y agoYeah, that could become legally awful. Which is appropriate; it appears $admin ruined a lot of people's days. I suspect that person will soon have a fair amount of time to decide on what new career to pursue to pay down the fines when they get out...
- redm 9y agoI suspect they assume it was an ex-admin probably because of timing. There may not be any proof to back up that claim and thus there may be no repercussions. Until they release specifics if ever, it's hard to know what to make of the status message.
- 3dm0nd 9y agoThey should be able to use https://github.com/hartator/wayback-machine-downloader https://github.com/hartator/wayback-machine-downloader and get at least a static version of the website back online.
- DKnoll 9y agoIf a VPS provider didn't manage to have even an offline backup of their own website they should dissolve the company.
- rurban 9y agoIf it was an ex-admin he might have deleted all the backups also. But they should be able to tell this publicly, otherwise it's similar to recent case where the newbie deleted the prod DB without any backup.
- foepys 9y agoIsn't this the time where you put stuff on tape and store it somewhere outside of physical reach? Is tape even used anymore?
- wlesieutre 9y agoYep, I'm not involved at all but I believe my workplace does its offsite backups on tape with a service like Iron Mountain http://www.ironmountain.com/Services/Data-Management/Tape-Vaulting.aspx http://www.ironmountain.com/Services/Data-Management/Tape-Va...
- tyingq 9y agoA patient disgruntled admin would have altered the backups to write garbage to the tapes for however long the rotation cycle is, then kicked off the production melee.
- _ph_ 9y agoThe parent poster talked about an offline backup. That is the big idea of offline (and even better, offsite) backups, that you cannot delete or destroy them from a computer, just by destroying the physical media.
- tanilama 9y agoDamn, they are fucked. How could people trust them any more???
- Sir_Substance 9y agoDick move from the ex-admin, but I'm curious to know what would compel an ex-employee to take such a brazenly criminal and traceable yet damaging action. I'd like to know more, I think...
- ktta 9y agoI think this one would qualify as a criminal move, not just a dick move
- deleted 9y ago[deleted]
- mikestew 9y agoWell, I’m guessing it starts with rationalization of why it is not criminal, and how you’re so smart that it won’t be traceable. Wrong on both counts, usually, but that’s my guess.
- jacquesm 9y agoThere is no excuse for it, no matter what the company did this affects the customers of the company just as hard as it affects the company. Likely some of those will go out of business, people will lose their jobs, go bankrupt.
- treyfitty 9y agoSo do whistle blowers. That line of reasoning doesn't set the bar for what's right and wrong. Are you sure no matter what the company did? What if the CEO threatened the ex-admins family? Or if the ex-admin found child porn on the CEOs computer? There's a fine line between right and wrong in most situations. The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.
- jacquesm 9y ago> So do whistle blowers. This is not a whistleblowing case, it is a case of wanton destruction by a former employee. > That line of reasoning doesn't set the bar for what's right and wrong. Dragging in all kinds of stuff that has no bearing on the case doesn't set the bar either. > Are you sure no matter what the company did? Yep. > What if the CEO threatened the ex-admins family? > Or if the ex-admin found child porn on the CEOs computer? In that case you go to the police and file a report with them. Hurting the company, the employees and customers when your target is the CEO is ineffective and illegal besides. > There's a fine line between right and wrong in most situations. No, it's crystal clear that this was wrong in any way you would like to look at it. > The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation. This is a criminal act, pure and simple. If the CEO did anything illegal this guy/girl is an idiot for doing something illegal himself.
- deleted 9y ago[deleted]
- zokier 9y agoWiped by ex-admin, or by the ineptitude of current admins that can't maintain proper exit procedure?
- Kenji 9y agoWho says he used his credentials and didn't leave and use a backdoor?
- tqkxzugoaupvwqr 9y agoThe current admins ineptitude did not wipe anything. Yes, with better procedures this could possibly have been prevented, but the current admins are not the malicious party.
- soared 9y agoMy most favorite of all razors is Hanlon's: "do not attribute to malice what can be explained by stupidity"
- epmatsw 9y agoNot sure how that's applicable here? Seems like an ex-admin deleting things is pretty clearly malicious...
- komali2 9y agoHe's saying the current admins are stupid for not investigating every possible route an ex admin could take for tanking the business.
- soared 9y agoDon't necesarrily attribute the error to the ex-admin, because the current admins were stupid enough to let it happen. Yes, the ex-admin is wrong and shouldn't have done it, but it is a crime of opportunity. If I leave $20 on the sidewalk the thief is wrong for stealing it, but it is partially my fault for being stupid enough to leave $20 on the ground.
- kashif 9y agoUse Vault from Hashicorp where possible.
- oneplane 9y agoThat would most likely have made 0 difference.
- thrillgore 9y agoIt's a process problem, not a product problem.
- wilhil 9y agoOther than treating staff well, how would you go about stopping something like this? As my own company is growing, we fully trust all employees, (limiting only what is essential), but, a dev ops guy if he was so inclined could technically do something like this... It always scares me.
- dmuth 9y agoSince the title says "Ex-Admin", revoking credentials when someone leaves the company is a must. If there's centralized auth (AD), this is pretty straightforward. If not, then at the very least it makes sense to use Puppet, Ansible, etc. to lock that person's account on all production servers. Beyond that, be sure to keep regular backups (and test them), and audit all user actions. (feed the logs into something like Splunk, running on a separate machine)
- tooltalk 9y agoI bet the ex-admin was also responsible for keeping backups and auditing/logging all access, in addition to managing pretty much everything else.
- swozey 9y agoThere really should be more systems that require 2 keys to do things like delete servers, load balancers, etc. I'm not aware of any. It's crazy to think that if an admins AWS/GCP/Azure account is compromised (or the cloud provider is compromised) a few commands/clicks to an API server could delete everything.
- benjohnson 9y agoWe swiped an idea from launching Nuclear Missiles: Typically you need two keys from two people to launch. For really important accounts - we have three people who each know two thirds of the password. It requires two people to then log in and do damage. For example if the root password was CatDogFish then Person1: CatDog_ Person2: _DogFish Person3: Cat_Fish Two people can then log in and watch what the other person is doing.
- jldugger 9y agoBeen waiting for a company to announce shutdown after this was posted: https://news.ycombinator.com/item?id=14476421 https://news.ycombinator.com/item?id=14476421 Possibly related?
- Dreami 9y agoWell it's kind of the opposite, isn't it? ;)
- fps 9y agoIn both cases, critical data with no backups was deleted by an ex-employee. In the "first day" story, it just happened to be both his first day and last day. I guess the difference is that one was a developer and the other an admin. No telling from this story if the person who deleted the data was an employee when they deleted the data.
- jldugger 9y agoIf you take Verelox's word for it. Or if you want to be more conspiratorial, you could see your firing on the wall, tweak the new hire docs to lead to trashing prod, and just wait for the inevitable. Or these two events are unrelated. Or the whole deleted prod on day 1 story is made up.
- emeraldd 9y agoAnecdotally, the timing doesn't match. Based on this https://www.lowendtalk.com/discussion/116329/what-s-up-with-verelox-being-down https://www.lowendtalk.com/discussion/116329/what-s-up-with-... it looks like they went down on early on the 8th give or take a bit and the other was posted around then 2nd ... Edit: Make that the 7th based on: https://www.facebook.com/Verelox/posts/1886196381643427?comment_id=1893011374295261&comment_tracking=%7B%22tn%22%3A%22R%22%7D https://www.facebook.com/Verelox/posts/1886196381643427?comm...
- pavement 9y agoI would expect to see some kind of police report, and prosecution of an individual charged with a crime, no?
- ntaylor 9y agoSeems like nothing short of a felony. I can't conceive why someone with half a brain would do this.
- deleted 9y ago[deleted]
- Cshelton 9y agoYes, it is criminal and the ex-admin will go to jail. The cases for this are pretty straightforward... not sure why someone would throw their life away by doing it...
- yaur 9y agoWell, whats actually not clear from their message is if the person who is currently an "ex administrator" was an ex-admin at the time that they deleted everything or if they were fired because of a (bad) mistake and are now an ex-admin.
- pavement 9y agoBeing fired for an honest error made in good faith is a misclassification of such a problem. If a single current, acting employee causes an outage by way of human error, then process and planning are to blame for the destruction. If there isn't a police report, and charges aren't pressed against a malicious individual, then the company may be at fault. Certainly the employee could willingly and pro-actively step down, out of personal guilt and feelings of shame, but then, one need not qualify with "ex" as simply "employee" will suffice. A criminal incident is worthy of such a clarification, indicating that the incident is a deliberate attack, but human error is not.
- 9y ago
- yangtheman 9y agoProper exit procedure should have disabled all access from this ex-admin..., unless s/he had some sort of cron job or launched some process that would execute commands at certain time? I am very curious to know how it was done.
- redm 9y agoThat's all true, but in reality, we often count on the better nature of people, goodwill, for a proper break on both sides. If someone is planning a malicious exit, it can be very hard to stop them depending on how "integrated" they are.
- weaksauce 9y agoIT admin people have the keys to the building and pretty much all data at the end of the day. Trust is everything and reputation is extremely important. This will not go well for the ex-admin one way or another either by lawsuit or blacklisting.
- superflyguy 9y agoWell, assuming he doesn't change his name and fake his employment history. Or just deny it. Or threaten to sue for libel if anyone claims it was him that did it. If it was me I'd claim they screwed up (restored a backup onto the backups, something like that, happens all the time) then blamed me. Let's be honest, they're more screwed than he is.
- idealogue 9y agoAnd yet, they are often abused, underpaid, and treated as replaceable. The only surprise should be that this doesn't happen far more often.
- weaksauce 9y agoAt the core, IT people are usually seen as cost centers and not revenue generators. Not that I disagree with the business owners a lot of the time because IT is usually not the thing that makes a lot of companies money.
- yardie 9y agoSo in addition to the criminal side of things I guess the ex-admin wants to work in manual labor or fast food. There is no way in hell he'd have the references or pass the background. BTW, we had a netadmin interview a few months ago. Guy was really smart, aced the technical and group interview. We were really looking forward to hiring him, and only needed to pass a background and reference check. HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.
- damm 9y agoBackground checks don't stop things like this. The problem is you have to hire trustworthy people in these positions. Regardless of what crime they may (or may not) have made; you are looking for the person and what they do. Not what they did. This is a common problem; we look at the past a bias of the future. Life only works out that way if the person is too unwilling to change; and that again is something you should look for in the hiring process. Lastly; hiring ex-hackers isn't a bad thing. Caring about a background check when hiring an ethical hacker or someone who turned their life around; only shames them and pushes them back where they came. So be careful or you only end up criminalizing being a criminal.
- yardie 9y agoSorry. If someone intentionally destroys company property I would want nothing to do with them, ever. This isn't a simple b&e where the perp rarely knows the victim. He was given the tools to do his job and intentionally used them to undermine his company and colleagues. Also, he is not a hacker, just an asshole. Having the admin login and password doesn't make you elite. The only weakness he exploited was himself.
- damm 9y agoBeing a criminal or commiting a crime has nothing to do with destroying company property. That's the problem with asking for people's background. What they did; and more importantly have they recovered and not commited another crime of similar circumstances. P.S. I knew a kid when I was younger who helped an ISP start up in Colorado Springs in the late 90's... he got fired and he hacked them and spent 6months in jail; eventually they re-hired him after he got out of jail only for a repeat offense. There is a point; he did it twice. The first time they could pass it off as they didn't know; the second time they hired him they were legally liable. So this gentleman should spend some time in jail for what he did.
- cannonpr 9y agoA lot of 'managed' hosting providers are pretty bad with security, there still is a major provider that just gives root credentials to all servers to all techs not just admins, doesn't audit who accesses which credentials, and doesn't rotate credentials, doesn't rate limit dumping credentials... That's before we go into more interesting issues with their security. Frankly I am surprised this sort of thing doesn't happen more often ? In some ways it both restores some of my faith in people while reducing some of it at the same time in a different vector.
- gaius 9y agoWho even knows it was an ex-admin? Could be the current one fat-fingered it and is trying to shift the blame! We just don't know.
- philfrasty 9y agoExactly. Mind-boggling how easy it is to present people a logical reason.
- pmarreck 9y agoThis is so stupid. If you have a problem with your employer, either you quit or they fire, you move on, full stop. If you're in a relationship and someone isn't happy enough with you and breaks up with you, the dignified response is NOT to key their car. I see employment relationships mostly the same way. Either it works (for both) or doesn't (for either or both ends). And having switched jobs quite a few times, the next one is always better for you, regardless.
- HipHopHacker 9y ago> I see employment relationships mostly the same way Apparently they did not. At the end of the day, the people working at the company are the ones who are doing the work, and who have control of the means of production. The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case. I worked at a Fortune 100 investment bank where this happened. Everyone knew layoffs were coming. One week after layoffs came, a digital "bomb" went off wrecking many servers. So security went through, trying to find evidence (nothing incriminating from what I heard, although they had a strong suspect) and also looking for more bombs. They missed out on finding and defusing one, because another one went off a month later. The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.
- jjnoakes 9y agoA person who can destroy servers is no more indispensable than an employee who can destroy a physical office.
- yjftsjthsd-h 9y agoYeah, a better case would be firing the admin then the site goes down and nobody can fix it.
- 9y ago
- jacquesm 9y agoThis is why you have the backups stored under a different account than the primaries and you make sure that nobody has access to both accounts.
- treyfitty 9y agoThis may be an unpopular opinion, but I want to preface this by saying: "before passing judgement, context is always necessary." Mario Savio was a Free Speach Activist and organized a protest to protect the Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he says "there's a time when the operation of the machine becomes so odious... that you can't take part... and you've got to indicate to the people in charge that unless you're free, the machine will be prevented from running at all!" Applied to free speech, this notion of disrupting the functioning of an organization was lauded, because freedom of speech is just that important. But let's shift to employment. Without employment, it's very hard to survive. And here's a situation where the people in charge has the upper hand in every arena- hiring, pay, work Place behavior... etc. How do we know that the ex-admin wasn't blackmailed by the CEO to come back to work for free to fix something, or future references will be negative? Why are we so quick to side with the employer in this matter when we know nothing of the situation at all? Why do we start calling the employee a felon? He hasn't even been charged yet. My point is, context is important. Fine, corporations have the power to ruin your life as a deterrent to keep you from acting against their interests, and that's just the way society is. And fine, We're not all rational at every instance of life. The calculus of establishing status quo equilibrium of those two conditions/constraints is hard, but without context to the situation, who are we to decide who's right or wrong? Would you label Mario Savio wrong for protesting and urging protestors to prevent the operation of the college from functioning in the name of preserving Free speech wrong? No, because you've learned the context.
- antfarm 9y agoI wonder whether the ex-admin was already an ex-admin at the time he wiped the servers.
- bobbob1921 9y agoLots of comments are interpreting "ex-admin" as someone who was fired and then after went and did this. Just want to float the possibility that "ex-admin" could also mean someone was employed there, then did this and is now no longer employed as a result of doing this. (Btw, IMO there is no excuse or justification for any admin or exadmin to ever do this. Among many other issues is the fact he deleted the data/work of individuals who had nothing to do with whatever "problem" he has with Verelox )
- sqldba 9y agoWhat if their information was all secretly being leaked to intelligence agencies and there was no way to ever let them know except to burn everything? There's probably excuses and justifications. I personally wouldn't do it and they're probably wrong for doing it but I don't want to jump to conclusions and moral absolutes so easily.
- stevenh 9y agoIf I ran a hosting company and all of my servers were compromised by ring -3 malware exploiting the Intel AMT vulnerability, the first thing I'd do is privately inform Intel that I intend to go public with the story and sue for damages, after which Intel would perhaps offer a very generous bribe for my silence and a week-long window to replace all of the server processors for free, on the one condition that I bury the truth by fabricating a story about an imaginary ex-employee who improbably was both smart enough to gain an administrative position in a large company while also being stupid enough to risk decades in prison for petty revenge over workplace drama.
- forgottenacc57 9y agoWeird. That's exactly what happened to the data centers at Area 51 too.
- ronilan 9y agoAlso similar to when Gillette paid a very generous bribe to bury Occam's and Hanlon's razors. /s
- sethrin 9y agoI'm sad both that I can't upvote this more than once and that I can't think of any way I'd ever get to steal this line.
- qb45 9y agoNo, these were stolen by aliens but CIA covered it up by pretending to be covering up AMT malware ;) Seriously though, would Verelox still be running unpatched AMT many weeks after the disclosure of this authentication bug? Or does GP think there are more bugs which Intel hopes to sweep under the rug forever by individually covering each incident? They would spend quite a money on these bribes while AMT bugs can simply be fixed with BIOS updates. This Intel conspiracy doesn't make sense. It's aliens, folks, I know it.
- midnitewarrior 9y ago
- quicksilver03 9y agoThe thread title should be changed to "Verelox allegedly wiped by ex-admin": we only know one side of the story.
- defkev 9y ago> Ex-admin commits social suicide just to be a dick Yeah, well, sounds awfully like one of thus "A hacker deleted all our customer data" fairy tales.
- svakacast 9y agoAnyone of us know what is the best way to get refunded? My Company lost 20.000€ for this joke.
- bm1362 9y agoLawsuit, most likely.
- tgtweak 9y agoThis exact thing happened to realitychecknetwork hosting about 6 years ago (now rebranded to serverstack and digitalocean). There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients. "Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared. They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean. They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it. Good cautionary tale for segregation of credentials and proper user key management.
- mintplant 9y agoLooks like DO has managed to keep this detail out of the record for the most part, save some forum posts that pop up if you search "realitychecknetwork digitalocean". It would explain the odd gap in time in their Wikipedia page history: > In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.
- tgtweak 9y agoYeah Ben and Moisey are both incredibly competent sysadmins and shrewd businessmen. They deserve the success they've had but they were very close to desperation after this incident. I can't blame them for omitting it from the records, the reality is likely that digital ocean and serverstack are both much more secure as a result.
- FussyZeus 9y agoProblem is at some point some person or more likely group of people has to be entrusted with the Master Keys. Using different keys for production machines and that sort of thing is all well and good but they're kept somewhere, meaning if an admin decides to screw you, there isn't a lot you can do. Seems like the better option is keep your admins happy as much as possible.
- tw04 9y agoIt's always interesting watching startups learn the lessons that thousands of enterprise learned along the way. "Why would you ever want offline tapes sitting in iron mountain, how inefficient". Nothing is foolproof, but anytime you've got constant network access to every last copy of your data, you're begging to lose it. It's the reason why people who think one copy (redundantly dispersed or not) in AWS S3 is sufficient scares me to death. Is it unlikely Amazon would get hacked and have the entire thing blown up? Sure... but if we go to war with China I wouldn't want to bet my company on it.
- nulagrithom 9y agoIf we go to war with China I don't think you or anyone else will care much about your company. I know you meant it as an example, but this sort of extreme attitude towards security is just another footgun.
- kstrauser 9y ago> If we go to war with China I don't think you or anyone else will care much about your company. Why? We went to war with Europe and Asia a few times and businesses kept chugging along here in the states.
- sabujp 9y agothat was before people had lots of nukes
- tw04 9y agoYou're assuming war will involve armed conflict from the start. If we go to war with China that will be the end of the war.
- DashRattlesnake 9y ago> If we go to war with China I don't think you or anyone else will care much about your company. Probably not in the immediate aftermath, but someone might decades later, if the company actually does something valuable.
- keithpeter 9y agohttps://www.lowendtalk.com/discussion/116329/what-s-up-with-verelox-being-down https://www.lowendtalk.com/discussion/116329/what-s-up-with-... Some posts from Verelox staff towards bottom third of this forum page search for user name Verelox
- deleted 9y ago[deleted]
- CM30 9y agoThis is why a hosting company needs to both segregate credentials to only what an employee needs for their job, as well as to revoke them the minute they leave the company. Otherwise while the vast majority of your staff will be decent people and not cause problems like this, it just takes one angry ex staff member with a grudge to cause problems. They also need to revise their backup system too. There should rarely if ever be a risk that any data is 'unrecoverable', yet their update says some data will just be impossible to get back. As for the employee involved... well I hope they like the inevitable lawsuit their selfish, stupid actions will bring them. I don't care what you think of a company you worked for, there's no excuse to destroy their business through actions like this. Also, good luck getting any jobs in the industry after too. Because with this on your track record, no one will touch you with a ten foot bargepole. So yeah, what a disaster all round.