5 ms·

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises so

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the tests be run on the production environment. There's bad luck and theres asking for trouble.

alextgordon 16y ago
If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.
- chesser 16y ago
  To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend all day and night making sure your repositories are secure, backed up and always available."
  - alextgordon 16y ago
    If we're going to use hindsight, we might as well look at the cause and the effect. Yes, they missed out some pretty obvious things. And what happened? A few hours of downtime because their restore was slow, plus a tiny bit of inconsequential data loss. Hardly a catastrophe. The fact is that every site has some sort of problems. Many of them will be completely obvious like this one. And while github could have gone through and attempted to fix them all, I much prefer they spend their time doing things that have more than a few hours of impact on my life. No doubt they will fix the issues involved today.
    - chesser 16y ago
      Your post is a good illustration of the type of attitude we want to scrupulously avoid when appointing someone to be in charge of a database.
  - risotto 16y ago
    The half a million users validates their techniques, no matter how much armchair quarterbacking. The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom. I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it very much is a race and features, for availability or recovery or other, are viciously prioritized and many things don't happen until something breaks. If you don't thing Github cuts the mustard, svn on Google Code probably won't have problems like this... Disclaimer: I'm personal friends with much of the Github crew.
    - chesser 16y ago
      > The half a million users validates their techniques, Let me introduce you to my friend, GeoCities. Funny thing about free hosting. > no matter how much armchair quarterbacking. Transactions aren't armchair quarterbacking. Binary backups aren't armchair quarterbacking. Separating development from production isn't armchair quarterbacking. Kindly, you have no idea what you're talking about. You can't give them credit for the repositories being mostly intact, when the ONLY parts that broke were the parts they mucked with to tie them into their database. > but the reality of startups is that it very much is a race and features, for availability or recovery And those are exactly the places where they screwed up.
      - risotto 16y ago
        Are you sitting in a chair? Do you not work at Github? Then you're armchair quarterbacking. We all are. Even if you are a quarterback for another team (I'm a DBA myself) Anyway, it's a great discussion so we can learn from other peoples mistakes. I will be triple checking my restores later today, and likely halt the project I'm working on to get cold standbys shored up asap. (but it's hard to prioritize house keeping over customer centric features in the race I'm running along side the Github crew)
        chesser 16y ago
        "a pejorative modifier to refer to a person who experiences something vicariously rather than first-hand, or to a casual critic who lacks practical experience" Ironically the most fitting application would be to say that they're armchair quarterbacking their own database administering.
- chesser 16y ago
  > If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now. Beta is the stage you reach AFTER you flip on transactions, institute and test quick restores, and separate dev from production.
jrockway 16y ago
So basically, you don't make mistakes, only other people do. Is it cold up there on that high horse?
- aaronblohowiak 16y ago
  It is trivial to put up barriers to mistakes.
  - risotto 16y ago
    After you make them, sure. But trying to predict mistakes before they happen can be difficult to impossible.
- binaryfinery 16y ago
  No I make mistakes all the time. That's kinda my point.
risotto 16y ago
I wonder if this is a rails thing. It's an out of the box pattern to put production, development and testing credentials into one database configuration file. Also from what I know Github is a flat org where all the devs have ability to do work in production. My company is like this too, and while it sometimes leads to scary mistakes, it also leads to massive productivity over having to go through a release engineering team.

alextgordon 16y ago

If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.

chesser 16y ago
To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend all day and night making sure your repositories are secure, backed up and always available."
- alextgordon 16y ago
  If we're going to use hindsight, we might as well look at the cause and the effect. Yes, they missed out some pretty obvious things. And what happened? A few hours of downtime because their restore was slow, plus a tiny bit of inconsequential data loss. Hardly a catastrophe. The fact is that every site has some sort of problems. Many of them will be completely obvious like this one. And while github could have gone through and attempted to fix them all, I much prefer they spend their time doing things that have more than a few hours of impact on my life. No doubt they will fix the issues involved today.
  - chesser 16y ago
    Your post is a good illustration of the type of attitude we want to scrupulously avoid when appointing someone to be in charge of a database.
- risotto 16y ago
  The half a million users validates their techniques, no matter how much armchair quarterbacking. The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom. I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it very much is a race and features, for availability or recovery or other, are viciously prioritized and many things don't happen until something breaks. If you don't thing Github cuts the mustard, svn on Google Code probably won't have problems like this... Disclaimer: I'm personal friends with much of the Github crew.
  - chesser 16y ago
    > The half a million users validates their techniques, Let me introduce you to my friend, GeoCities. Funny thing about free hosting. > no matter how much armchair quarterbacking. Transactions aren't armchair quarterbacking. Binary backups aren't armchair quarterbacking. Separating development from production isn't armchair quarterbacking. Kindly, you have no idea what you're talking about. You can't give them credit for the repositories being mostly intact, when the ONLY parts that broke were the parts they mucked with to tie them into their database. > but the reality of startups is that it very much is a race and features, for availability or recovery And those are exactly the places where they screwed up.
    - risotto 16y ago
      Are you sitting in a chair? Do you not work at Github? Then you're armchair quarterbacking. We all are. Even if you are a quarterback for another team (I'm a DBA myself) Anyway, it's a great discussion so we can learn from other peoples mistakes. I will be triple checking my restores later today, and likely halt the project I'm working on to get cold standbys shored up asap. (but it's hard to prioritize house keeping over customer centric features in the race I'm running along side the Github crew)
      - chesser 16y ago
        "a pejorative modifier to refer to a person who experiences something vicariously rather than first-hand, or to a casual critic who lacks practical experience" Ironically the most fitting application would be to say that they're armchair quarterbacking their own database administering.
chesser 16y ago
> If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now. Beta is the stage you reach AFTER you flip on transactions, institute and test quick restores, and separate dev from production.

jrockway 16y ago

So basically, you don't make mistakes, only other people do. Is it cold up there on that high horse?

aaronblohowiak 16y ago
It is trivial to put up barriers to mistakes.
- risotto 16y ago
  After you make them, sure. But trying to predict mistakes before they happen can be difficult to impossible.
binaryfinery 16y ago
No I make mistakes all the time. That's kinda my point.

risotto 16y ago

I wonder if this is a rails thing. It's an out of the box pattern to put production, development and testing credentials into one database configuration file. Also from what I know Github is a flat org where all the devs have ability to do work in production. My company is like this too, and while it sometimes leads to scary mistakes, it also leads to massive productivity over having to go through a release engineering team.