7 ms·

How Facebook Ships Code

129 points by stunr69 14y ago

gouranga 14y ago
What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be. I'd never put my data near FB. They are simply irresponsible.
- TazeTSchnitzel 14y ago
  It's a little irresponsible, but how else do you expect them to debug live code?
  - bnr 14y ago
    You don't. Reproduce the issue on a development instance.
    - damncabbage 14y ago
      I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)
      - TazeTSchnitzel 14y ago
        Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
        mgkimsal 14y ago
        A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports). Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L. Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.
        wpietri 14y ago
        I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..." Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume. Madness, sure, but common madness.
    - rimantas 14y ago
      How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?
      - Goladus 14y ago
        For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.
  - arethuza 14y ago
    "how else do you expect them to debug live code" I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence). Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log. I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks. [NB Of course, none of this would be applicable to Facebook]
  - gouranga 14y ago
    Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire. It just works. In the last 5 years, we've never hooked a debugger to production instance or given access to a developer. In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP). That's how it's done.
    - TazeTSchnitzel 14y ago
      You're right. Now that I think about it, I doubt Facebook is that careful, considering how often they break things.
    - rimantas 14y ago
      No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.
      - alttab 14y ago
        By being able to design software that doesn't need or allow crazy data configurations. My whole mantra is don't manage complexity: avoid it.
        gouranga 14y ago
        You nailed it - thanks for the absolutely spot on comment. Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.
      - gioele 14y ago
        Boundary value analysis? Machine state transitions testing? Equivalence partitioning? There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.
- jschuur 14y ago
  Note that it doesn't say whether they all have write access, or whether all the data they have access to is encrypted in some way. And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.
  - gouranga 14y ago
    Either way, it simply doesn't matter.
- piggity 14y ago
  You finance and billing guys will never understand. These are rockstars. They work for facebook. They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';
  - jacques_chester 14y ago
    Of course not. Much faster to type UPDATE users SET email = username || '@facebook.com'; Then let the users sort out the exceptions.
    - alttab 14y ago
      I think he was being sarcastic - you can see the semicolon in the middle of the command which looks like pretty much what they did with the whole e-mail debacle. You simply removed the second incomplete statement.
      - jacques_chester 14y ago
        I now realise that I am underqualified to work at Facebook. :(
  - deleted 14y ago
    [deleted]
  - getsat 14y ago
    >implying this was an engineering team decision
- jdsemrau 14y ago
  >I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be. Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.
  - yashchandra 14y ago
    "they should follow the same legal requirements" Perhaps. But they certainly do not have the restrictions of financial audit requirements and things like Dodd Frank etc. which is overwhelming all the major banks right now and is a big way to earn money for consultants if they know ABC of audit/compliance in 2012. Anyway, the point is that fb even though not ideal in how they store/use user data (which I personally am not a fan of as well), it does not matter as much as it matters to a bank/financial institution. I say this while I work for clients that are major banks.
- jorgeleo 14y ago
  But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed. One size (of engineering practices) fit all mentality is not the correct approach either.
  - gouranga 14y ago
    You are correct - it is not a financial organsiation. It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc. A list of investments or your mortgage statement is way less important.
- gaius 14y ago
  Right. FB doesn't get that all they have is the user experience, and it is bad user experience if I drop out of a conversation I'm having because their DBA-free database has decided to simply, silently drop my last comment. Or if it takes me several attempts to post a photo. Or if I invite someone to an event, and they never get the invite. You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.
  - darkarmani 14y ago
    > Grown-up companies have full-time professional DBAs I don't know if I'd call them "professional," but they definitely have full-time DBAs. ;)
    - mkjones 14y ago
      What do you mean? I've worked with some of our DBAs, and they're quite good. In fact, I can't think of a single site issue that was caused by a DBA. I work on fighting spam at FB, and we make use of mysql quite a bit.
michaelmartin 14y ago
I really like seeing this approach listed: "resourcing for projects is purely voluntary. A PM lobbies group of engineers, tries to get them excited about their ideas. Engineers decide which ones sound interesting to work on." That sounds exactly the same as how Github's engineers work. It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. And if you can't get anyone interested in working on the project, then it's a good indicator it may not be a project worth completing for the company anyway. I'm sure there are times when someone has to say "We need someone to do this", but I'd be curious to hear from someone who works in one of these environments how common an issue that really is.
- moondistance 14y ago
  I'm curious - what happens when an engineer wants to propose an idea? Do they need to get a PM on-board to lead the project? Who decides if the idea is good? The PM/engineer divide feels uncomfortable to me, but I have limited experience with these roles, so it's likely simply for lack of knowledge. I hope someone can clarify this for me.
  - michaelmartin 14y ago
    From the article at least, it seems like there aren't a whole lot of PMs at Facebook. If an engineer has an idea, they seem to be more encouraged to just find some people to work with, build it, and then roll it out to a very small section of Facebook's users and see how they react to it. Good ideas might then be taken further, but bad ones (Or I guess, ones people just don't care about) won't be. (I don't work there though, so I have no way of knowing if that's right or not, sorry!)
    - wilfra 14y ago
      From the article: 1/10 PM's to Engineers and 500 engineers = 50 PM's. That's not an entirely small amount for Facebook since they don't have a zillion different products and features that need to be owned, like a company like Microsoft or Google.
- 14y ago
jsvaughan 14y ago
Previously on HN: http://news.ycombinator.com/item?id=2594083 http://news.ycombinator.com/item?id=2594083 ↪ How Facebook _actually_ pushes updates to the site I came across this originally on the Etsy dev blog, rather than HN, and that particular post had some good other stuff about Flickr and Etsy: http://codeascraft.etsy.com/2011/06/01/pushing-facebook-flickr-etsy/ http://codeascraft.etsy.com/2011/06/01/pushing-facebook-flic...
Silhouette 14y ago
Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems that are always changing, so users can never rely on anything working the same way from one visit to the next. I see organisations with access to sensitive personal data being cavalier at best about how they handle it. I see a major blunder every few weeks that at best causes serious irritation to users and at worst risks significant loss of business and/or legal/regulatory consequences. And I see huge brands whose egotistical staff don't realise that they are successful despite evidently not bothering with either real product management or robust testing, not because of these things, and who don't seem to notice that they are still relying almost entirely on momentum from one or two really big successes from early on to maintain their user base and revenues.
- jrockway 14y ago
  I wouldn't lump Google in there with Facebook. We have very strict controls on access to user data (we can't even see email addresses in logs), and we have not adopted the motto "move fast and break things". We do extensive automated testing and have release processes that are designed to minimize problems in the event of a bad push. Yes, bugs happen from time to time, but it's software -- there is no known practical technique to produce bug-free software, so we have to settle for mostly-bug-free software instead. This isn't being amateurish or egotistical, it's being realistic.
  - Silhouette 14y ago
    I'm not arguing for unrealistic quality levels, and I will acknowledge immediately that my experience could be atypical. However, in fairness, if I look at all of the software that I use regularly in a professional capacity today, then it is clearly Google products that are the most buggy, and by a very wide margin. For example, I have a client who uses Google Docs/Drive. We have rarely managed to hold a meeting without one of our small team struggling to see either a word processor document or a spreadsheet properly, and that's just the minor cosmetic or browser incompatibility bugs that keep appearing along with all those minor UI changes. We have also experienced some much more serious problems, including corruption/data loss and seeing the entire change history for some files become inaccessible for no apparent reason. In other words, it's not just minor UI errors that crept in as the product evolved, there are evidently fundamental flaws in the underlying architecture that don't store the data robustly. Another example: I spent a couple of days last week trying to figure out why a site that had been working fine for users until recently and had not been changed at all on our side was suddenly generating bug reports. It turned out that recent Chrome builds have broken some HTML5 features in multiple ways. There have been related bug reports in some cases, but they have been closed as the specific example given no longer seemed to be a problem. Again, the nature of the problems makes it obvious that these are not just one-liner issues but fundamental flaws, typically where Chrome is so aggressive with its cache usage and/or trigging redrawing/relayout that it just plain doesn't work. And even though the bugs had been reported, obviously the correct root cause was never identified and fixed. Another example: I recently spent some time looking into how the Closure Tools are coming along. Have you tried the examples/demonstrations for the Closure Library recently? Many of them simply don't work in Gecko-based browsers, and that would be obvious if anyone working on the project had made even a cursory attempt to test them for five minutes. I will finish by once again acknowledging that my experiences here may be atypical, and that the particular projects I've mentioned that I happen to be using may not reflect the wider Google culture. But on the evidence before me, I see an organisation that keeps breaking things in its rush to push new features out and that demonstrably lacks robust architectures that keep data safe, effective testing processes before pushing code into production, and proper issue resolution processes when defects do get reported.
its_so_on 14y ago
This is the real Facebook secret sauce in convenient flowchart form: What's the most evil thing we can think of doing? V Candidate <-----<------- Think of the next-most evil thing we can do V ^ Can we code that? (no) ------->------->^ V (yes) ^ Is it legal? ^ V (yes/no) ^ Can we get away with it? (abs. not)--->^ V (yes/maybe/prob. not) ^ Keep shipping! ^ V ^ Did we get in trouble for it? (no) ^ V (yes) V ^ Claim it was a mistake! V ^ V V ^ Still in trouble? (no)-> Keep feature->^ V (kind of) ^ Being sued for it? (no) --->--->^ V (yes) ^ Throw money at lawsuit ^ V ^ Did we lose? (yes/no) ----->--->^
- alttab 14y ago
  Not very constructive to the conversation, but I gotta give it to ya, this is pretty funny.
- gouranga 14y ago
  That's great - thanks for posting it :)
  - gouranga 14y ago
    If you're going to down vote, please at least say why...
    - bajsejohannes 14y ago
      Comments on HN tends to be voted down if they are not substantial. To quote http://ycombinator.com/newswelcome.html http://ycombinator.com/newswelcome.html: Does your comment teach us anything? That page also says that a simple thanks may be acceptable, but the consensus seems to be that those comments are superfluous as well.
- 14y ago
DigitalSea 14y ago
Don't get me wrong a developer-led company in theory sounds like an awesome idea because the developers know the product better than any product manager ever could, but that isn't a good thing. I've worked with companies that have ultra-strict testing of features, where many eyes see and use the code before being pushed out and while that process works it can get in the way of progress when the politics come out to play. The whole move fast, make mistakes mantra is a bad approach when you're a high-trafficked site like Facebook that risks jeopardising your revenue. A second of down time can be very costly. Knowingly pushing out half or completely untested features might be acceptable if you have a small user base, you're a new Internet startup or your logo says alpha or beta, but certainly not an established brand and company it's stupid. This is all my own opinion, of course.
- alttab 14y ago
  I would argue the opposite. Facebook is in prime position to move quickly and break things. They have enough traction, gravity, and brand recognition they can afford to make mistakes. If you have a small user base or are just getting started, delivering the best product experience is the #1 thing you can do for success. Is iterating faster and making more mistakes and giving the early adopters (ones that champion your service the most to others) a sub-par experience worth the speed? Sometimes, yes. All the time? No. Facebook can change your privacy settings, steal your e-mail address, recognize you automatically in photos, watch your browsing activity outside of their walled garden, and sell your data to advertisers. And yet they still have 500million+ users.
sreyaNotfilc 14y ago
This may be my favorite quote... "Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves." I actually like this idea. Building my own website as well as working as the senior engineer during my day job forces me to be involved in all facets of web developing. The jobs are not abstracted. You are expected to know what you're doing from the front end to the back end. If not, nothing get done in a timely manner. I like this method because the front end is tightly woven in with the business logic of the module that you're working on. In other words, you know what the code is doing inside and out.
- alttab 14y ago
  I agree. Specialists are really just full stack engineers who haven't fulfilled their potential. Why hire a self-proclaimed UX guy? Unless I have all the amazing engineers I need for my team (99% not the case), I'm looking for someone who can do multiple things. The best ones are ones that can do schema design, query design, system and model architecture, front-end wireframes, javascript, and implementation. That said, your comment about the front end being "tightly woven in with the business logic of the module" scares me a little. The front end is presentation and interaction. Unless most of your businesses mojo is client side, even a full stack engineer should design each layer as fully cohesive and encapsulated components. I may have misunderstood you, however.
  - sreyaNotfilc 14y ago
    What I mean by "tightly woven" is that you know exactly what values/components/types need to communicate with each other whether its the front end or the back end. You're not wasting time trying to match a spec sheet or fulfill a precondition to an existing back-end procedure (cause honestly those things just don't fit right in). That is, you're not spending time trying to put a square peg into a round hole.
epriest 14y ago
See this question on Quora for some clarifications from people who are or have been Facebook employees (including myself). The article (particularly the original version) has a lot of inaccuracies, and is now around 18 months out of date. http://www.quora.com/Facebook-Engineering/How-accurate-is-the-How-Facebook-Ships-Code-article-written-by-yeeguy http://www.quora.com/Facebook-Engineering/How-accurate-is-th...
yawgmoth 14y ago
I like the idea of encouraging a high-performance culture, but I don't think the 'perform or die' atmosphere would be healthy for many engineers. I know, idolizing 'rockstar programmers' is a sort of new hotness and I understand that a company like Facebook wants to have super-talented developers, but developers grow and learn new tricks as they mature, and they might take more than six months to do so.
- alttab 14y ago
  True - but those engineers don't work at Facebook. This isn't no-child left behind. This isn't hand-hold time. This is the most expensive and expansive internet application in the world. If they need time to ramp up, they can do it on someone else's product and come to Facebook when they're ready.
  - dot 14y ago
    the most expensive internet application? what do you mean?
  - wpietri 14y ago
    That sounds impressively macho, but it's an attitude that has long-term organizational costs. HN just had a great article on how Microsoft's internal competition deeply harmed the company. The number one complaint I hear from departing Google engineers is the absurd internal promotions system. Years ago I did a gig at eBay, and I thought their macho attitude was a giant source of problems. Plenty of good, sane people were driven off (or driven mad) by artificially high-pressure situations. Every email about a promotion mentioned how somebody had worked all night to get something done; they were promoting more for drama than for skill. And a "this isn't hand-hold time" attitude was common among senior technical staff, which meant that people often hid their weaknesses rather than getting the help they needed. The lesson I learned from that is that software companies that take normal circumstances with the intensity of emergencies are gradually cutting their own throats. That's a lesson reinforced for me spending a lot of time with a family member in hospitals last year. Even when survival was on the line, the best doctors and nurses proceeded with patience and kindness, working to train staff and improve systems as they went. In an actual emergency they moved like it was an emergency. But only then. If they can be serene and thoughtful while dealing with brain tumors, I don't think there's any reason that people at Facebook have to puff themselves up with self-importance.
Goladus 14y ago
The facebook system actually sounds really solid, especially the "boot camp" thing that so many companies fail to have, however it's probably pretty expensive. Choosing between fast, cheap, and good, Facebook is choosing fast and good. For many, cheap and good but slow is more desirable. One of the risks to consider with a "devops" oriented approach is that you may become more dependent on it than you want. Often, applications split things into a few different categories depending on how tweakable they need to be. There's code, configuration, app administration, and data. Code shouldn't need to change often. Configuration may need to change when the environment changes. App administration (eg creating new accounts) needs to change often, and data is always changing (or at least growing). The risk is that developers will design the system so that only developers can administer it. Configuration, the settings that may need to be tweaked by sys admins long after the original developers have left the project, may wind up in the code or sometimes lumped into the database alongside end-user options. It's not a reason not to take this approach, just something to consider when developing internal processes and culture.
five_star 14y ago
"very engineering driven culture. ”product managers are essentially useless here.” is a quote from an engineer. engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime" Maybe this is why Facebook seems to be chaotic for users. They change and change to whatever design they wanted without much consideration about what the user's would feel about the design. Facebook has now become the combined features of the other existing social media.
sodelate 14y ago
what does this mean?
peapicker 14y ago
I stopped reading at "very unique". It is unique, or it isn't. Intensifiers to 'unique' tell me the writing is below par; and I've been correct about this enough over the years that I stopped bothering.
Dybbuk 14y ago
Well, I'll be joining Facebook in a few weeks. I am a bit of a laid back type and don't know if I fit into their culture of moving fast.
krosaen 14y ago
""" Engineers responsible for testing, bug fixes, and post-launch maintenance of their own work. there are some unit-testing and integration-testing frameworks available, but only sporadically used. """ Sounds like a lot of code debt accumulating that could bite them hard down the road - it's one thing to write and manually verify bug free code, it's another for a different engineer to make sure he/she doesn't break that code inadvertently a year later when the original author has moved on to another project or company. I'm not talking about 100% test coverage; if the smoke test for a feature breaking is someone noticing while playing with the site, in the long run it strikes me as a much less efficient way to verify and fix regressions than using an automated test suite. Writing good tests is hard, but keeping a product bug free as more and more functionality accumulates without automated test suites is even harder in my experience.
- mkjones 14y ago
  This article's about a year and a half old. We have pretty good unit test coverage on a good chunk of our code (especially core stuff), though admittedly not everything. Some groups put particular emphasis on this (e.g. the messages team is great about testing), and it shows in the reliability of their products. Even better, they end up building frameworks that make it easier for the rest of engineering to write tests, and drive the whole ecosystem forward.
  - krosaen 14y ago
    Good to hear. Related: a good article by Eric Ries on how in many situations within a startup technical debt can be used effectively. http://www.startuplessonslearned.com/2009/07/embrace-technical-debt.html http://www.startuplessonslearned.com/2009/07/embrace-technic...

129 points by stunr69 14y ago

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be. I'd never put my data near FB. They are simply irresponsible.

TazeTSchnitzel 14y ago
It's a little irresponsible, but how else do you expect them to debug live code?
- bnr 14y ago
  You don't. Reproduce the issue on a development instance.
  - damncabbage 14y ago
    I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)
    - TazeTSchnitzel 14y ago
      Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
      - mgkimsal 14y ago
        A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports). Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L. Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.
        wpietri 14y ago
        I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..." Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume. Madness, sure, but common madness.
  - rimantas 14y ago
    How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?
    - Goladus 14y ago
      For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.
- arethuza 14y ago
  "how else do you expect them to debug live code" I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence). Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log. I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks. [NB Of course, none of this would be applicable to Facebook]
- gouranga 14y ago
  Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire. It just works. In the last 5 years, we've never hooked a debugger to production instance or given access to a developer. In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP). That's how it's done.
  - TazeTSchnitzel 14y ago
    You're right. Now that I think about it, I doubt Facebook is that careful, considering how often they break things.
  - rimantas 14y ago
    No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.
    - alttab 14y ago
      By being able to design software that doesn't need or allow crazy data configurations. My whole mantra is don't manage complexity: avoid it.
      - gouranga 14y ago
        You nailed it - thanks for the absolutely spot on comment. Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.
    - gioele 14y ago
      Boundary value analysis? Machine state transitions testing? Equivalence partitioning? There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.
jschuur 14y ago
Note that it doesn't say whether they all have write access, or whether all the data they have access to is encrypted in some way. And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.
- gouranga 14y ago
  Either way, it simply doesn't matter.
piggity 14y ago
You finance and billing guys will never understand. These are rockstars. They work for facebook. They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';
- jacques_chester 14y ago
  Of course not. Much faster to type UPDATE users SET email = username || '@facebook.com'; Then let the users sort out the exceptions.
  - alttab 14y ago
    I think he was being sarcastic - you can see the semicolon in the middle of the command which looks like pretty much what they did with the whole e-mail debacle. You simply removed the second incomplete statement.
    - jacques_chester 14y ago
      I now realise that I am underqualified to work at Facebook. :(
- deleted 14y ago
  [deleted]
- getsat 14y ago
  >implying this was an engineering team decision
jdsemrau 14y ago
>I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be. Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.
- yashchandra 14y ago
  "they should follow the same legal requirements" Perhaps. But they certainly do not have the restrictions of financial audit requirements and things like Dodd Frank etc. which is overwhelming all the major banks right now and is a big way to earn money for consultants if they know ABC of audit/compliance in 2012. Anyway, the point is that fb even though not ideal in how they store/use user data (which I personally am not a fan of as well), it does not matter as much as it matters to a bank/financial institution. I say this while I work for clients that are major banks.
jorgeleo 14y ago
But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed. One size (of engineering practices) fit all mentality is not the correct approach either.
- gouranga 14y ago
  You are correct - it is not a financial organsiation. It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc. A list of investments or your mortgage statement is way less important.
gaius 14y ago
Right. FB doesn't get that all they have is the user experience, and it is bad user experience if I drop out of a conversation I'm having because their DBA-free database has decided to simply, silently drop my last comment. Or if it takes me several attempts to post a photo. Or if I invite someone to an event, and they never get the invite. You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.
- darkarmani 14y ago
  > Grown-up companies have full-time professional DBAs I don't know if I'd call them "professional," but they definitely have full-time DBAs. ;)
  - mkjones 14y ago
    What do you mean? I've worked with some of our DBAs, and they're quite good. In fact, I can't think of a single site issue that was caused by a DBA. I work on fighting spam at FB, and we make use of mysql quite a bit.

michaelmartin 14y ago

I really like seeing this approach listed: "resourcing for projects is purely voluntary. A PM lobbies group of engineers, tries to get them excited about their ideas. Engineers decide which ones sound interesting to work on." That sounds exactly the same as how Github's engineers work. It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. And if you can't get anyone interested in working on the project, then it's a good indicator it may not be a project worth completing for the company anyway. I'm sure there are times when someone has to say "We need someone to do this", but I'd be curious to hear from someone who works in one of these environments how common an issue that really is.

moondistance 14y ago
I'm curious - what happens when an engineer wants to propose an idea? Do they need to get a PM on-board to lead the project? Who decides if the idea is good? The PM/engineer divide feels uncomfortable to me, but I have limited experience with these roles, so it's likely simply for lack of knowledge. I hope someone can clarify this for me.
- michaelmartin 14y ago
  From the article at least, it seems like there aren't a whole lot of PMs at Facebook. If an engineer has an idea, they seem to be more encouraged to just find some people to work with, build it, and then roll it out to a very small section of Facebook's users and see how they react to it. Good ideas might then be taken further, but bad ones (Or I guess, ones people just don't care about) won't be. (I don't work there though, so I have no way of knowing if that's right or not, sorry!)
  - wilfra 14y ago
    From the article: 1/10 PM's to Engineers and 500 engineers = 50 PM's. That's not an entirely small amount for Facebook since they don't have a zillion different products and features that need to be owned, like a company like Microsoft or Google.
14y ago

jsvaughan 14y ago

Previously on HN: http://news.ycombinator.com/item?id=2594083 http://news.ycombinator.com/item?id=2594083 ↪ How Facebook _actually_ pushes updates to the site I came across this originally on the Etsy dev blog, rather than HN, and that particular post had some good other stuff about Flickr and Etsy: http://codeascraft.etsy.com/2011/06/01/pushing-facebook-flickr-etsy/ http://codeascraft.etsy.com/2011/06/01/pushing-facebook-flic...

Silhouette 14y ago

Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems that are always changing, so users can never rely on anything working the same way from one visit to the next. I see organisations with access to sensitive personal data being cavalier at best about how they handle it. I see a major blunder every few weeks that at best causes serious irritation to users and at worst risks significant loss of business and/or legal/regulatory consequences. And I see huge brands whose egotistical staff don't realise that they are successful despite evidently not bothering with either real product management or robust testing, not because of these things, and who don't seem to notice that they are still relying almost entirely on momentum from one or two really big successes from early on to maintain their user base and revenues.

jrockway 14y ago
I wouldn't lump Google in there with Facebook. We have very strict controls on access to user data (we can't even see email addresses in logs), and we have not adopted the motto "move fast and break things". We do extensive automated testing and have release processes that are designed to minimize problems in the event of a bad push. Yes, bugs happen from time to time, but it's software -- there is no known practical technique to produce bug-free software, so we have to settle for mostly-bug-free software instead. This isn't being amateurish or egotistical, it's being realistic.
- Silhouette 14y ago
  I'm not arguing for unrealistic quality levels, and I will acknowledge immediately that my experience could be atypical. However, in fairness, if I look at all of the software that I use regularly in a professional capacity today, then it is clearly Google products that are the most buggy, and by a very wide margin. For example, I have a client who uses Google Docs/Drive. We have rarely managed to hold a meeting without one of our small team struggling to see either a word processor document or a spreadsheet properly, and that's just the minor cosmetic or browser incompatibility bugs that keep appearing along with all those minor UI changes. We have also experienced some much more serious problems, including corruption/data loss and seeing the entire change history for some files become inaccessible for no apparent reason. In other words, it's not just minor UI errors that crept in as the product evolved, there are evidently fundamental flaws in the underlying architecture that don't store the data robustly. Another example: I spent a couple of days last week trying to figure out why a site that had been working fine for users until recently and had not been changed at all on our side was suddenly generating bug reports. It turned out that recent Chrome builds have broken some HTML5 features in multiple ways. There have been related bug reports in some cases, but they have been closed as the specific example given no longer seemed to be a problem. Again, the nature of the problems makes it obvious that these are not just one-liner issues but fundamental flaws, typically where Chrome is so aggressive with its cache usage and/or trigging redrawing/relayout that it just plain doesn't work. And even though the bugs had been reported, obviously the correct root cause was never identified and fixed. Another example: I recently spent some time looking into how the Closure Tools are coming along. Have you tried the examples/demonstrations for the Closure Library recently? Many of them simply don't work in Gecko-based browsers, and that would be obvious if anyone working on the project had made even a cursory attempt to test them for five minutes. I will finish by once again acknowledging that my experiences here may be atypical, and that the particular projects I've mentioned that I happen to be using may not reflect the wider Google culture. But on the evidence before me, I see an organisation that keeps breaking things in its rush to push new features out and that demonstrably lacks robust architectures that keep data safe, effective testing processes before pushing code into production, and proper issue resolution processes when defects do get reported.

its_so_on 14y ago

This is the real Facebook secret sauce in convenient flowchart form: What's the most evil thing we can think of doing? V Candidate <-----<------- Think of the next-most evil thing we can do V ^ Can we code that? (no) ------->------->^ V (yes) ^ Is it legal? ^ V (yes/no) ^ Can we get away with it? (abs. not)--->^ V (yes/maybe/prob. not) ^ Keep shipping! ^ V ^ Did we get in trouble for it? (no) ^ V (yes) V ^ Claim it was a mistake! V ^ V V ^ Still in trouble? (no)-> Keep feature->^ V (kind of) ^ Being sued for it? (no) --->--->^ V (yes) ^ Throw money at lawsuit ^ V ^ Did we lose? (yes/no) ----->--->^

alttab 14y ago
Not very constructive to the conversation, but I gotta give it to ya, this is pretty funny.
gouranga 14y ago
That's great - thanks for posting it :)
- gouranga 14y ago
  If you're going to down vote, please at least say why...
  - bajsejohannes 14y ago
    Comments on HN tends to be voted down if they are not substantial. To quote http://ycombinator.com/newswelcome.html http://ycombinator.com/newswelcome.html: Does your comment teach us anything? That page also says that a simple thanks may be acceptable, but the consensus seems to be that those comments are superfluous as well.
14y ago

DigitalSea 14y ago

Don't get me wrong a developer-led company in theory sounds like an awesome idea because the developers know the product better than any product manager ever could, but that isn't a good thing. I've worked with companies that have ultra-strict testing of features, where many eyes see and use the code before being pushed out and while that process works it can get in the way of progress when the politics come out to play. The whole move fast, make mistakes mantra is a bad approach when you're a high-trafficked site like Facebook that risks jeopardising your revenue. A second of down time can be very costly. Knowingly pushing out half or completely untested features might be acceptable if you have a small user base, you're a new Internet startup or your logo says alpha or beta, but certainly not an established brand and company it's stupid. This is all my own opinion, of course.

alttab 14y ago
I would argue the opposite. Facebook is in prime position to move quickly and break things. They have enough traction, gravity, and brand recognition they can afford to make mistakes. If you have a small user base or are just getting started, delivering the best product experience is the #1 thing you can do for success. Is iterating faster and making more mistakes and giving the early adopters (ones that champion your service the most to others) a sub-par experience worth the speed? Sometimes, yes. All the time? No. Facebook can change your privacy settings, steal your e-mail address, recognize you automatically in photos, watch your browsing activity outside of their walled garden, and sell your data to advertisers. And yet they still have 500million+ users.

sreyaNotfilc 14y ago

This may be my favorite quote... "Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves." I actually like this idea. Building my own website as well as working as the senior engineer during my day job forces me to be involved in all facets of web developing. The jobs are not abstracted. You are expected to know what you're doing from the front end to the back end. If not, nothing get done in a timely manner. I like this method because the front end is tightly woven in with the business logic of the module that you're working on. In other words, you know what the code is doing inside and out.

alttab 14y ago
I agree. Specialists are really just full stack engineers who haven't fulfilled their potential. Why hire a self-proclaimed UX guy? Unless I have all the amazing engineers I need for my team (99% not the case), I'm looking for someone who can do multiple things. The best ones are ones that can do schema design, query design, system and model architecture, front-end wireframes, javascript, and implementation. That said, your comment about the front end being "tightly woven in with the business logic of the module" scares me a little. The front end is presentation and interaction. Unless most of your businesses mojo is client side, even a full stack engineer should design each layer as fully cohesive and encapsulated components. I may have misunderstood you, however.
- sreyaNotfilc 14y ago
  What I mean by "tightly woven" is that you know exactly what values/components/types need to communicate with each other whether its the front end or the back end. You're not wasting time trying to match a spec sheet or fulfill a precondition to an existing back-end procedure (cause honestly those things just don't fit right in). That is, you're not spending time trying to put a square peg into a round hole.

epriest 14y ago

See this question on Quora for some clarifications from people who are or have been Facebook employees (including myself). The article (particularly the original version) has a lot of inaccuracies, and is now around 18 months out of date. http://www.quora.com/Facebook-Engineering/How-accurate-is-the-How-Facebook-Ships-Code-article-written-by-yeeguy http://www.quora.com/Facebook-Engineering/How-accurate-is-th...

yawgmoth 14y ago

I like the idea of encouraging a high-performance culture, but I don't think the 'perform or die' atmosphere would be healthy for many engineers. I know, idolizing 'rockstar programmers' is a sort of new hotness and I understand that a company like Facebook wants to have super-talented developers, but developers grow and learn new tricks as they mature, and they might take more than six months to do so.

alttab 14y ago
True - but those engineers don't work at Facebook. This isn't no-child left behind. This isn't hand-hold time. This is the most expensive and expansive internet application in the world. If they need time to ramp up, they can do it on someone else's product and come to Facebook when they're ready.
- dot 14y ago
  the most expensive internet application? what do you mean?
- wpietri 14y ago
  That sounds impressively macho, but it's an attitude that has long-term organizational costs. HN just had a great article on how Microsoft's internal competition deeply harmed the company. The number one complaint I hear from departing Google engineers is the absurd internal promotions system. Years ago I did a gig at eBay, and I thought their macho attitude was a giant source of problems. Plenty of good, sane people were driven off (or driven mad) by artificially high-pressure situations. Every email about a promotion mentioned how somebody had worked all night to get something done; they were promoting more for drama than for skill. And a "this isn't hand-hold time" attitude was common among senior technical staff, which meant that people often hid their weaknesses rather than getting the help they needed. The lesson I learned from that is that software companies that take normal circumstances with the intensity of emergencies are gradually cutting their own throats. That's a lesson reinforced for me spending a lot of time with a family member in hospitals last year. Even when survival was on the line, the best doctors and nurses proceeded with patience and kindness, working to train staff and improve systems as they went. In an actual emergency they moved like it was an emergency. But only then. If they can be serene and thoughtful while dealing with brain tumors, I don't think there's any reason that people at Facebook have to puff themselves up with self-importance.

Goladus 14y ago

The facebook system actually sounds really solid, especially the "boot camp" thing that so many companies fail to have, however it's probably pretty expensive. Choosing between fast, cheap, and good, Facebook is choosing fast and good. For many, cheap and good but slow is more desirable. One of the risks to consider with a "devops" oriented approach is that you may become more dependent on it than you want. Often, applications split things into a few different categories depending on how tweakable they need to be. There's code, configuration, app administration, and data. Code shouldn't need to change often. Configuration may need to change when the environment changes. App administration (eg creating new accounts) needs to change often, and data is always changing (or at least growing). The risk is that developers will design the system so that only developers can administer it. Configuration, the settings that may need to be tweaked by sys admins long after the original developers have left the project, may wind up in the code or sometimes lumped into the database alongside end-user options. It's not a reason not to take this approach, just something to consider when developing internal processes and culture.

five_star 14y ago

"very engineering driven culture. ”product managers are essentially useless here.” is a quote from an engineer. engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime" Maybe this is why Facebook seems to be chaotic for users. They change and change to whatever design they wanted without much consideration about what the user's would feel about the design. Facebook has now become the combined features of the other existing social media.

sodelate 14y ago

what does this mean?

peapicker 14y ago

I stopped reading at "very unique". It is unique, or it isn't. Intensifiers to 'unique' tell me the writing is below par; and I've been correct about this enough over the years that I stopped bothering.

Dybbuk 14y ago

Well, I'll be joining Facebook in a few weeks. I am a bit of a laid back type and don't know if I fit into their culture of moving fast.

krosaen 14y ago

""" Engineers responsible for testing, bug fixes, and post-launch maintenance of their own work. there are some unit-testing and integration-testing frameworks available, but only sporadically used. """ Sounds like a lot of code debt accumulating that could bite them hard down the road - it's one thing to write and manually verify bug free code, it's another for a different engineer to make sure he/she doesn't break that code inadvertently a year later when the original author has moved on to another project or company. I'm not talking about 100% test coverage; if the smoke test for a feature breaking is someone noticing while playing with the site, in the long run it strikes me as a much less efficient way to verify and fix regressions than using an automated test suite. Writing good tests is hard, but keeping a product bug free as more and more functionality accumulates without automated test suites is even harder in my experience.

mkjones 14y ago
This article's about a year and a half old. We have pretty good unit test coverage on a good chunk of our code (especially core stuff), though admittedly not everything. Some groups put particular emphasis on this (e.g. the messages team is great about testing), and it shows in the reliability of their products. Even better, they end up building frameworks that make it easier for the rest of engineering to write tests, and drive the whole ecosystem forward.
- krosaen 14y ago
  Good to hear. Related: a good article by Eric Ries on how in many situations within a startup technical debt can be used effectively. http://www.startuplessonslearned.com/2009/07/embrace-technical-debt.html http://www.startuplessonslearned.com/2009/07/embrace-technic...