19 ms·

Upgrading GitHub from Rails 3.2 to 5.2

510 points by masnick 8y ago

mooreds 8y ago
Super cool explanation of some of the real world difficulties of upgrading large rails applications. Really liked the transparency around process and timeline, as well as the lessons learned section.
hartator 8y ago
> Upgrade early and upgrade often Not sure about the upgrade early. It’s a different kind of pain to be one of the first to use a new Rails version vs lagging a couple of months behind.
- mperham 8y ago
  Waiting a few months is good. It's a reasonable strategy to wait for x.y.1 but Rails 5.2.0 was so stable that 5.2.1 took ~6 months to appear.
- deleted 8y ago
  [deleted]
- gsnedders 8y ago
  I think there's two sane options: * track master, and notice breakage as soon as possible (you get much smaller deltas when trying to figure out breakage, which is always a big help), and are able to fix it or report it upstream ASAP, or * hold off slightly till the new release has had more bugs found post-release.
  - hartator 8y ago
    Main issue is more with other gems that need to be updated as well. It may take take months. You can't do that on your own.
    - lostapathy 8y ago
      You can't do them all, but you can certainly pitch in and submit patches. Outside rspec-related gems, upgrading usually isn't that hard. And like all open source, somebody has to do it.
    - craigmcnamara 8y ago
      Fork, use your fork, send a pull request. Almost every rails upgrade I make forks and send patches on gems that I need that I forgot about. I get good responsiveness and merge rates. Most of the time a gem needs to be able to support multiple rails versions, so in that case I usually send pull request setting up the Appraisal gem to test for appropriate compatibility.
- komali2 8y ago
  Yes we tried this once, with webpack 4, and realized in about an hour that like half our packages were incompatible and had patches weeks out. I'm talking big packages, less-loader. Htmlwepbackplugin still requires @next for webpack4 to work right.
- tetha 8y ago
  I agree, at least for anything that's hard to test and has some inertia, like a larger code base. For example most of our configuration management is setup to just pull in the latest cookbooks from upstream during tests, and as long as all integration tests across all projects succeed, they get uploaded to our chef-server. People argued that it would be annoying because things would break all the time. And yeah, things break with updates, though the opscode community is remarkably disciplined about semver. But that's what we have tests for. And honestly: I'd rather deal with one broken update per day than 300 broken updates once per year. One bad update usually requires some nudging and that's it. 300 bad updates at once are a fully blown nightmare and you'll need days just to figure out what is even going on.
- titusjohnson 8y ago
  Yeah, being a first adopter can be a pain, but I don't think that's what they're saying here. Github was 4 years and 2 major versions behind on upgrades. Upgrade Early does not mean Upgrade Instantly. I would argue that "lagging a couple of months behind" is upgrading early, because these upgrade horror stories always come out of companies that are years and years behind. Having some patience and waiting a little, combined with the discipline to not wait too long is part of a mature skillset.
- Rapzid 8y ago
  I'm not sure how GitHub can say they "learned" this yet. Wasn't their last rails upgrade also a huge ordeal and a blog post? How about get back to us in a few years and we'll see if you learned anything this time around.
- craigmcnamara 8y ago
  The Appraisal gem is excellent for ensuring compatibility across many Rails and Ruby versions. Adding it to a project with a good test suite is a pretty good guarantee of an easy upgrade.
sigzero 8y ago
I had no idea that Github even used Rails. The things I learn.
- powersurge360 8y ago
  If you didn't know that then let me share with you this interesting story from 2012. I'm going to repeat it from memory so my details may be a little fuzzy but I'll include a link which should tell the story more faithfully. So back in 2012 rails had a default behavior where you could mass assign values from a POST to a user and there wasn't any scrubbing of that, by default. Someone realized this was a Bad Idea and issued a pull request that would have fixed it. Instead of accepting the PR, DHH (I think it was him) said something along the lines of 'competent programmers would not leave that setting in place' and rejected the PR. The exploit discoverer thought about this and tried it against github, which was known to run on rails and the code worked! From there he was able to manipulate the permissions on github to get access to the rails repo where he reopened and accepted his own pull request. He was promptly banned. https://gist.github.com/peternixey/1978249 https://gist.github.com/peternixey/1978249
  - dyeje 8y ago
    This was a huge deal at the time, here's one of the HN threads. https://news.ycombinator.com/item?id=3663197 https://news.ycombinator.com/item?id=3663197
    - tspike 8y ago
      Thanks for sharing, that was very entertaining.
  - orf 8y ago
    Where is the actual pull request?
    - powersurge360 8y ago
      Re-reading the material from that era I think I embellished a little in my memory. I think there was a pull request but he didn't reopen and accept it. Sounds like he just pushed a commit that said something along the lines of "why can I commit this to master". I'm busy at work so I can't dig in but I'm sure someone will find that original PR. If not tonight I'll see if I can't find it. EDIT: Went ahead and found it. It was an issue. https://github.com/rails/rails/issues/5228 https://github.com/rails/rails/issues/5228 EDIT2: It looks like DHH may have even gone so far as to delete his comments in this issue. There's folks referencing him and one side of a conversation in places. Pretty funny.
chris72205 8y ago
Similar post from Shopify about a year ago on their experience upgrading from Rails 4.2 to 5 https://shopifyengineering.myshopify.com/blogs/engineering/upgrading-shopify-to-rails-5-0 https://shopifyengineering.myshopify.com/blogs/engineering/u...
- deleted 8y ago
  [deleted]
- tim333 8y ago
  And some similar discussion on "Shopify now on Rails 5.0. started 12 years ago on 0.5, the First version released" https://news.ycombinator.com/item?id=13448219 https://news.ycombinator.com/item?id=13448219
dyeje 8y ago
Wow they've been running 3.2 for this long? That's wild considering the talk Eileen gave at RailsConf this year made it sound like alot of the Rails 6 scalability stuff was based on GitHub's existing work.
- rst 8y ago
  They've been running 3.2 with local monkeypatches -- which is part of the reason that upgrading was problematic. (Though certainly not all; over that span, there were lots of breaking changes to supported and documented APIs.)
dasil003 8y ago
No mention of the rails:update rake task, which is a very valuable tool to get your boilerplate updated. I'm guessing at GitHub there is so much customization over the years that they wouldn't get much out of it, but it's still a valuable exercise to run through, and it's worthwhile to keep your boilerplate aligned as much as possible since it makes gems, documentation, and everything else more likely to align with shared community experience. Also, I want to add a big proviso to the lesson "Upstream your tooling instead of rolling your own". Historically in ruby, and now even moreso in node, the ease of pushing new packages and the trendiness of the language at times has led to a lot of churn and abandoning of packages. The trick is to include stable dependencies, and that requires quite a bit of experience and tea-leaf reading to do right. Often times maintaining your own stripped down internal lib can be a performance and maintenance cost win over including a larger batteries-included lib that ends up being poorly supported over time. For example, a lot of people got burned by using the state_machine gem that at one time was very hot and actively maintained but went on to get left in an aggressive limbo (https://github.com/pluginaweek/state_machine https://github.com/pluginaweek/state_machine).
- caseyf 8y ago
  > Also, I want to add a big proviso to the lesson "Upstream your tooling instead of rolling your own". I feel like this bit needed more of an explanation about how this applied to GitHub. If I were to write a post about working in a 10 year old Ruby codebase I'd definitely include "Kill your dependencies" as a bullet point.
  - tcopeland 8y ago
    > I'd definitely include "Kill your dependencies" as a bullet point Or at least your monkeypatches!
  - marcus_holmes 8y ago
    Yeah, I don't really understand about this, especially the security aspect of Gems. Every piece of externally-maintained code is a security risk, surely? You are implicitly trusting the maintainer of that Gem to not hide bad things in their code. And every Gem that they depend on. If the Gems are old and the maintainer is unpaid and doing other stuff, how sure can you be that they're still vetting all contributions for security? Or that they haven't handed over the maintenance to someone you no longer trust? Or that the maintainer hasn't succumbed to economic pressure and included some malicious code in their Gem? Or do you have to manually review every single line of code in every dependency yourself? That seems like a lot of work... I would definitely prefer to write my own code for a feature than review 1000's of sloc of someone else's code to spot any problems. I get that the core Rails codebase gets security-reviewed regularly, but does that happen for Gems? And is it methodical and thorough, or is it just "lots of eyeballs"? And if so, is there a threshold of Gem popularity below which there aren't enough eyeballs to spot problems and the Gem should be considered insecure? And if you do spot a problem, do you report it and hope the maintainer has time to do something about it? Or do you write a PR and submit it, hoping they accept it? Doesn't that then mean you're maintaining someone else's code base? Again, I would massively prefer to write and maintain my own code than maintain someone's else code (or wait for them to fix a problem that they may no longer care about). How do you build a secure application for something as trusted as Github while gleefully incorporating all this third-party code?
aantix 8y ago
Using the conditional boot loading, aren’t there structural differences in ActiveRecord queries/scopes that would run under 3.2 but not 5.2? Did GH just rewrite those scopes in their respective models and maintain a ton of if/else blocks for the different versions? And if so, didn’t they run into issues without the code not being DRY, e.g. someone fixes a 3.2 query, but not the corresponding 5.2 version?
- barrkel 8y ago
  If you have your test code unified, and have multiple CI pipelines, it should show up immediately on your build servers.
deleted 8y ago
[deleted]
stevebmark 8y ago
There are several "we upgraded Rails, it was huge, risky, and took months to years" blog posts from medium to large companies. I personally take this as a warning against using Rails. Ruby is one of the most dangerous dynamic languages to refactor, I don't see how struggling to do it for over a year is a selling point of the framework. It also feels counter to Rails's mantra of delivering value fast with little effort, until you need to upgrade, then you have months of no business value delivery and need to bring in experts to help.
- majewsky 8y ago
  > Ruby is one of the most dangerous dynamic languages to refactor A thousand times this. It's so much easier to do breaking changes and refactors in a language that's supporting you, instead of working against you.
  - cknoxrun 8y ago
    Dynamic language or not, having good test coverage will be your biggest support.
    - weberc2 8y ago
      It's definitely important, but we (Python shop) have "good" (85%) test coverage and we still see 500s in prod every day because of things a type checker could trivially catch. And this is just in the course of normal operation; this isn't even a migration. Having extensive experience with both Go and Python, I would conservatively estimate that Go requires ~30% fewer tests than untyped Python for the same confidence. That's more than 30% time savings; not only are you not writing 30% of the tests, but that's 30% fewer tests to have to maintain. Of course these aren't the only considerations--for certain tasks Python may be faster to develop with (although I think people forget about things like deployment, tooling, dependency management, performance requirements, etc when they make their estimations).
      - jb3689 8y ago
tjpnz 8y ago
It's scarily common for organizations to be running ancient versions of Rails in production. At my last gig we spent six months upgrading a Rails 2.3 application to 3.2 and before that I was working with a team that was maintaining an application written in Rails 1x. Kudos to GitHub for sharing this, I really hope they do future posts going into more detail. In my experience one of the hardest aspects to upgrading Rails is that so much of the really useful information has either fallen out of Google or succumbed to link rot.
- cptskippy 8y ago
  It's not just Rails, this common with many frameworks and even turnkey solutions from vendors. Nothing will ever meet your needs 100% and so you end up with some customization. It really depends on how integral those customizations were and how tightly they are coupled to the product being used.
- stouset 8y ago
  This is true of literally everything. People don't upgrade their dependencies across the board, and it's a massive problem for long-term security and maintainability.
  - jb3689 8y ago
    My current company doesn't even lock most of their dependencies. At first I thought it was crazy (and it is crazy) but it does mean we address compatibility issues immediately. We are mostly running background jobs though so it's safer than anything customer facing
    - stouset 8y ago
      My opinions on this have changed a lot over the past few years. At this point, I think anything that pins dependencies to specific versions is asking for long-term maintainability nightmares. Updates of your dependencies, operating system, language version, etc. should happen weekly, and any instance where you have to pin a dependency to an old version (e.g., a major version release that has some compatibility issues) should be dealt with ASAP. Obviously this isn't a tenable position in every circumstance, but I think it should be the default. Particularly in a world where the vast majority of security fixes go without an announcement or CVE.
throwaway427 8y ago
Given its maturity and settled place in the programming landscape it's always nice to see that Rails can still evoke irrational disdain in HN comments.
- pwelch 8y ago
  100%
- gameswithgo 8y ago
  Why is irrational to hate languages that are orders of magnitude slower than are necessary?
  - jb3689 8y ago
    Something-something IO something-something
  - goatlover 8y ago
    It's irrational to hate without mentioning tradeoffs. Sure, if performance is your only metric, then Ruby is a bad choice. But that's rarely the case, particularly with the web.
  - rpeden 8y ago
    That's a reasonable question to ask. I think in general, there are lots of reasons to like a language outside of its runtime performance. I love working with Go and Rust due to their performance. Any I work every day in C#, which ends up nice and quick, too. But I still love Ruby due to its expressiveness, and the way it works just seems to align with the way I think. But that's poetically because I used Smalltalk in the past and I like the bits of it that Ruby borrowed. :) To answer to original question, though. I'd say it's irrational to hate languages that are slower than necessary because it's irrational to hate a programming language at all. No matter what language it is, it's just a bunch of words on a screen. Use the ones you like and don't waste any brain cycles thinking about the ones you don't. Unless you're locked in a cube farm and forced to write Cobol at gunpoint all day. Hate might be rational then.
  - 8y ago
k__ 8y ago
I had the impression GH switched Ruby for Scala years ago.
- marksomnian 8y ago
  That was Twitter.
  - k__ 8y ago
    Oh, lol. Thanks :)
Twirrim 8y ago
> Upgrade early and upgrade often It seems daft to keep seeing this lesson being learned by tech companies, and keep seeing blog posts where most of the pain would have been handled easily by just making upgrading a key feature. Instead, tech managers and engineers seem to make the same mistakes over and over again, delaying those upgrades, until suddenly they discover it's a hard task to upgrade. I get delaying to _some_ degree, it's better to let other people figure out those sharp bits on the bleeding edge for you, but you need to set an explicit target for upgrading. At another large tech company I worked for, it took the security team swinging the sledgehammer to get teams to upgrade from known-vulnerable versions of Ruby on Rails. When they came to do it, they discovered the changes were so extreme that the effort involved in migrating was likely more than the effort involved in a complete re-write (they did at least have pretty comprehensive tests)
- cortesoft 8y ago
  It is easy to say that in the abstract, but you always have a finite time/resource budget for doing work. Effort spent upgrading is effort not spent doing other important work. Is the other work more important in the long run? The answer is not trivial to answer. This is why we call it 'tech debt'.. it is just like any other debt. You take it on because you don't have the current resources to avoid it, and you calculate that it is worth taking it on. But then, you are carrying the interest on it, and if you aren't careful it will grow to be unmanageable, and all your dev effort goes into just paying the interest without paying the principle.
  - Twirrim 8y ago
    It absolutely matters. Security is a feature. If you see upgrading as merely technical debt, you're never going to give it the appropriate attention.
    - regularfry 8y ago
      To twist the metaphor slightly, it's technical inflation.
    - 8y ago
gameswithgo 8y ago
How much money in server costs and how much electricity could be saved if Github didn't use an interpreted language, but something like Go, C#, F#, Java etc?
- jb3689 8y ago
  Github would not have been the same in any of those. They really took to some of the Rails concepts - a lot better than most Rails companies - and it shows in their product (routing, object structures, etc)
  - gameswithgo 8y ago
    So your claim is you could not create the same user experience in any language that is jitted or compiled? I can't really take that seriously.
    - jb3689 8y ago
      Sure, in retrospect you could create the same thing. It's just various text processing at the end of the day. What I'm claiming is that their choice of Rails led to certain choices which were really transparent throughout the product and are still there. It would've grown to be something totally different on another technology, so I don't think it's fair to just look at cost and performance. They did many things "the Rails way"
- erokar 8y ago
  If they'd used Java they would still be working on the prototype.
  - gameswithgo 8y ago
    I've seen quite a few attempts to measure productivity differences between different languages and there is not a consistent win being shown by dynamic languages in general. Perhaps ruby on rails is especially productive for the web, and maybe especially so when github.com launched, but there are lots of options now with similar productivity and 1 or 2 orders of magnitude better performance.
dwb 8y ago
Upgrading early(ish) and often, the very obvious preventative measure against terrible and failure-prone rewrite or upgrade projects, is one of the first things that falls by the wayside in the mostly short-termist logic that seems to dominate modern capitalism. It's absolutely infuriating.
conroy 8y ago
> The upgrade started out as kind of a hobby; engineers would work on it when they had free time. There was no dedicated team. I’m not sure why this still surprises me. For a company the size of Github, there should most certainly be a team responsible for these type of upgrades.
nautilus12 8y ago
Why in the world is github still on rails?
- notriddle 8y ago
  Why not?
- stephenhuey 8y ago
  Maybe because their codebase still serves their use cases very well? And perhaps they have little to gain and possibly much to lose if they ditch it? You didn't say much in your question, so I don't know if you feel they ought to rewrite with a popular SPA framework or use something like Elixir Phoenix, but if their Rails-based solution handily serves 30 million users, why do you feel so strongly they should move to something else?
  - mrdoops 8y ago
    Nothing wrong with Rails, especially if the team knows it well. Time to develop is the real cost in software most of the time. If Github wanted to integrate a lot of real-time features, then Elixir + Phoenix can't be beat. Depending on what they replace, a 10x in performance and a fraction of the servers needed is a nice win.
    - innocentoldguy 8y ago
      I've seen performance boosts closer to 20x when I've helped companies rewrite their Rails products in Elixir. I've also seen a reduction in server costs. In all fairness though, simply rewriting the Rails app in Rails, with the benefit of hindsight, probably would have resulted in a performance gain too.
      - ksec 8y ago
        I would bet $10 the scaling problem with Github would properly have more to deal with Git than Rails itself. Switching to Elixir wouldn't really help.
config_yml 8y ago
What do they mean by off-hours? I imagine on a global site like github, there are hardly off-hours?
- dyeje 8y ago
  Just because it's a global site doesn't mean the traffic is distributed uniformly across the day. Certain regions are going to have higher traffic during business hours. I'd guess their off hours are somewhere around 6pm PST when North / South America has stopped working, Europe / Africa is asleep, and India is just waking up.
- toasterlovin 8y ago
  They probably mean when they weren't tied up shipping features or tracking down bugs.
jimnotgym 8y ago
The regular thread of people piling in to criticise dynamic languages. Instead perhaps people could suggest a better language/framework that is more productive than Rails, and has had a long lifespan in a large codebase?
- BjoernKW 8y ago
  Although arguably not as productive Java fares pretty well in that regard, particularly when it comes to maintainability. Still, Rails is terrific. If it wasn’t for Rails Java today wouldn’t be as productive either.
  - alexhutcheson 8y ago
    Yeah Java isn't nearly as bad as it used to be, and it's pretty seamless to write your business logic in Kotlin if Java's verbosity annoys you.
    - AlphaSite 8y ago
      Honestly Java is getting better at a good pace now. It’s nowhere near as bad as the java 6 days.
- radium3d 8y ago
  I was surprised to learn that github runs on ruby on rails. Interesting, I didn't know Ruby has been around just as long as PHP. I still would choose PHP if my opinion matters, just from my experience with the slow performance of ruby on rails when I gave it a go just a few years back. PHP - 23 years Ruby - 23 years Java - 23 years JavaScript (nodejs) - 22 years (9 years) https://en.wikipedia.org/wiki/Dynamic_programming_language#Examples https://en.wikipedia.org/wiki/Dynamic_programming_language#E...
  - bal00ns 8y ago
    Don't forget Python - 28 years
jbergstroem 8y ago
As a comparison, here's gitlab's journey (issue opened March, 2016): https://gitlab.com/gitlab-org/gitlab-ce/issues/14286 https://gitlab.com/gitlab-org/gitlab-ce/issues/14286 Looks like the first scheduled milestone was 9.5 (a year ago) and the current is set for 11.4 (next release).
lunaru 8y ago
For my team, this article comes at some interesting timing, since we're bumping into some of the same issues with Rails. Rails is now a mature framework and part of the problem is its lack of consideration for large existing codebases running in production. While there are nice tools to help migrate (e.g. rails:update) that hit surface issues, the deep problem is that there are a lot of decisions made going from version to version that are obviously unfriendly to established projects. e.g.: https://github.com/rails/rails/issues/27231 https://github.com/rails/rails/issues/27231 Additionally, there are a lot of gems that are losing momentum, which are near-core to Rails. e.g.: https://github.com/thiagopradi/octopus/issues/490 https://github.com/thiagopradi/octopus/issues/490. This is a side effect of the above issue, where the alternatives to Rails are taking a lot of the community away to focus on newer/shiner things. Fortunately, we have companies like GitHub and Shopify that are still very much invested in the success of the ecosystem. All that said, it's still a great framework to go from 0 to production with a new idea or project. Other ecosystems we're entrenched in (Node for example) have their share of issues as well, but we won't go into those.
- lostapathy 8y ago
  > art of the problem is its lack of consideration for large existing codebases running in production To be fair - this has gotten a lot better. Upgrading 2.3 -> 3.2 was terrible. 3.2 -> 4.0 was terrible. 4.0 -> 4.1 was rough. Since then, I've found the upgrades pretty easy - to the point I ran rails 5.2.0-rc's in production for a while. As you fairly note, a big problem is that related gems lose momentum and they don't get updated - which blocks other updates. On the flip side, they usually aren't that hard to update and submit a PR on, either. Even not having those patches merged quickly is not so bad in ruby - it's easy to tell bundler to look at your fork of a gem on github rather than pulling the upstream.
  - bhaak 8y ago
    What was your problem with going from 3.2 to 4.0? I think the upgrade path from 2.3 being terrible is generally accepted as being true. But I don't remember any hard problems from 3.x onward. At least for Rails itself. Gems dependencies are another problem. Like at one project we had pains every time Rails upgraded their minor version because the previous devs thought using Squeel instead of the builtin ActiveRecord a good idea. Just for being able to write slightly "nicer" queries and now this is a major stopper for going to Rails 5.
ryenus 8y ago
Anyone knows which ruby runtime GitHub uses? Ruby MRI or JRuby etc.?
exabrial 8y ago
The advice at the end sounds exactly like something I'd say to someone going from 1.8 to 11 with Java. Great advice for any platform, very interesting to see the same conclusions from a totally different stack
starefossen 8y ago
As a person working for a large software consultancy in Scandinavia I hate to see so many using type safety as an excuse for not writing tests. At least a dynamic language forces you to write tests and frankly it is often easier to write tests in a dynamic language imho.
ksec 8y ago
I am very much looking forward to Rails 6.0 and see what Github / Shopify will upstream. Actually Instacart has lot of great gems too which I wish would have been the default solution in Rails.

510 points by masnick 8y ago

Super cool explanation of some of the real world difficulties of upgrading large rails applications. Really liked the transparency around process and timeline, as well as the lessons learned section.

hartator 8y ago

> Upgrade early and upgrade often Not sure about the upgrade early. It’s a different kind of pain to be one of the first to use a new Rails version vs lagging a couple of months behind.

mperham 8y ago
Waiting a few months is good. It's a reasonable strategy to wait for x.y.1 but Rails 5.2.0 was so stable that 5.2.1 took ~6 months to appear.
deleted 8y ago
[deleted]
gsnedders 8y ago
I think there's two sane options: * track master, and notice breakage as soon as possible (you get much smaller deltas when trying to figure out breakage, which is always a big help), and are able to fix it or report it upstream ASAP, or * hold off slightly till the new release has had more bugs found post-release.
- hartator 8y ago
  Main issue is more with other gems that need to be updated as well. It may take take months. You can't do that on your own.
  - lostapathy 8y ago
    You can't do them all, but you can certainly pitch in and submit patches. Outside rspec-related gems, upgrading usually isn't that hard. And like all open source, somebody has to do it.
  - craigmcnamara 8y ago
    Fork, use your fork, send a pull request. Almost every rails upgrade I make forks and send patches on gems that I need that I forgot about. I get good responsiveness and merge rates. Most of the time a gem needs to be able to support multiple rails versions, so in that case I usually send pull request setting up the Appraisal gem to test for appropriate compatibility.
komali2 8y ago
Yes we tried this once, with webpack 4, and realized in about an hour that like half our packages were incompatible and had patches weeks out. I'm talking big packages, less-loader. Htmlwepbackplugin still requires @next for webpack4 to work right.
tetha 8y ago
I agree, at least for anything that's hard to test and has some inertia, like a larger code base. For example most of our configuration management is setup to just pull in the latest cookbooks from upstream during tests, and as long as all integration tests across all projects succeed, they get uploaded to our chef-server. People argued that it would be annoying because things would break all the time. And yeah, things break with updates, though the opscode community is remarkably disciplined about semver. But that's what we have tests for. And honestly: I'd rather deal with one broken update per day than 300 broken updates once per year. One bad update usually requires some nudging and that's it. 300 bad updates at once are a fully blown nightmare and you'll need days just to figure out what is even going on.
titusjohnson 8y ago
Yeah, being a first adopter can be a pain, but I don't think that's what they're saying here. Github was 4 years and 2 major versions behind on upgrades. Upgrade Early does not mean Upgrade Instantly. I would argue that "lagging a couple of months behind" is upgrading early, because these upgrade horror stories always come out of companies that are years and years behind. Having some patience and waiting a little, combined with the discipline to not wait too long is part of a mature skillset.
Rapzid 8y ago
I'm not sure how GitHub can say they "learned" this yet. Wasn't their last rails upgrade also a huge ordeal and a blog post? How about get back to us in a few years and we'll see if you learned anything this time around.
craigmcnamara 8y ago
The Appraisal gem is excellent for ensuring compatibility across many Rails and Ruby versions. Adding it to a project with a good test suite is a pretty good guarantee of an easy upgrade.

sigzero 8y ago

I had no idea that Github even used Rails. The things I learn.

powersurge360 8y ago
If you didn't know that then let me share with you this interesting story from 2012. I'm going to repeat it from memory so my details may be a little fuzzy but I'll include a link which should tell the story more faithfully. So back in 2012 rails had a default behavior where you could mass assign values from a POST to a user and there wasn't any scrubbing of that, by default. Someone realized this was a Bad Idea and issued a pull request that would have fixed it. Instead of accepting the PR, DHH (I think it was him) said something along the lines of 'competent programmers would not leave that setting in place' and rejected the PR. The exploit discoverer thought about this and tried it against github, which was known to run on rails and the code worked! From there he was able to manipulate the permissions on github to get access to the rails repo where he reopened and accepted his own pull request. He was promptly banned. https://gist.github.com/peternixey/1978249 https://gist.github.com/peternixey/1978249
- dyeje 8y ago
  This was a huge deal at the time, here's one of the HN threads. https://news.ycombinator.com/item?id=3663197 https://news.ycombinator.com/item?id=3663197
  - tspike 8y ago
    Thanks for sharing, that was very entertaining.
- orf 8y ago
  Where is the actual pull request?
  - powersurge360 8y ago
    Re-reading the material from that era I think I embellished a little in my memory. I think there was a pull request but he didn't reopen and accept it. Sounds like he just pushed a commit that said something along the lines of "why can I commit this to master". I'm busy at work so I can't dig in but I'm sure someone will find that original PR. If not tonight I'll see if I can't find it. EDIT: Went ahead and found it. It was an issue. https://github.com/rails/rails/issues/5228 https://github.com/rails/rails/issues/5228 EDIT2: It looks like DHH may have even gone so far as to delete his comments in this issue. There's folks referencing him and one side of a conversation in places. Pretty funny.

chris72205 8y ago

Similar post from Shopify about a year ago on their experience upgrading from Rails 4.2 to 5 https://shopifyengineering.myshopify.com/blogs/engineering/upgrading-shopify-to-rails-5-0 https://shopifyengineering.myshopify.com/blogs/engineering/u...

deleted 8y ago
[deleted]
tim333 8y ago
And some similar discussion on "Shopify now on Rails 5.0. started 12 years ago on 0.5, the First version released" https://news.ycombinator.com/item?id=13448219 https://news.ycombinator.com/item?id=13448219

dyeje 8y ago

Wow they've been running 3.2 for this long? That's wild considering the talk Eileen gave at RailsConf this year made it sound like alot of the Rails 6 scalability stuff was based on GitHub's existing work.

rst 8y ago
They've been running 3.2 with local monkeypatches -- which is part of the reason that upgrading was problematic. (Though certainly not all; over that span, there were lots of breaking changes to supported and documented APIs.)

dasil003 8y ago

No mention of the rails:update rake task, which is a very valuable tool to get your boilerplate updated. I'm guessing at GitHub there is so much customization over the years that they wouldn't get much out of it, but it's still a valuable exercise to run through, and it's worthwhile to keep your boilerplate aligned as much as possible since it makes gems, documentation, and everything else more likely to align with shared community experience. Also, I want to add a big proviso to the lesson "Upstream your tooling instead of rolling your own". Historically in ruby, and now even moreso in node, the ease of pushing new packages and the trendiness of the language at times has led to a lot of churn and abandoning of packages. The trick is to include stable dependencies, and that requires quite a bit of experience and tea-leaf reading to do right. Often times maintaining your own stripped down internal lib can be a performance and maintenance cost win over including a larger batteries-included lib that ends up being poorly supported over time. For example, a lot of people got burned by using the state_machine gem that at one time was very hot and actively maintained but went on to get left in an aggressive limbo (https://github.com/pluginaweek/state_machine https://github.com/pluginaweek/state_machine).

caseyf 8y ago
> Also, I want to add a big proviso to the lesson "Upstream your tooling instead of rolling your own". I feel like this bit needed more of an explanation about how this applied to GitHub. If I were to write a post about working in a 10 year old Ruby codebase I'd definitely include "Kill your dependencies" as a bullet point.
- tcopeland 8y ago
  > I'd definitely include "Kill your dependencies" as a bullet point Or at least your monkeypatches!
- marcus_holmes 8y ago
  Yeah, I don't really understand about this, especially the security aspect of Gems. Every piece of externally-maintained code is a security risk, surely? You are implicitly trusting the maintainer of that Gem to not hide bad things in their code. And every Gem that they depend on. If the Gems are old and the maintainer is unpaid and doing other stuff, how sure can you be that they're still vetting all contributions for security? Or that they haven't handed over the maintenance to someone you no longer trust? Or that the maintainer hasn't succumbed to economic pressure and included some malicious code in their Gem? Or do you have to manually review every single line of code in every dependency yourself? That seems like a lot of work... I would definitely prefer to write my own code for a feature than review 1000's of sloc of someone else's code to spot any problems. I get that the core Rails codebase gets security-reviewed regularly, but does that happen for Gems? And is it methodical and thorough, or is it just "lots of eyeballs"? And if so, is there a threshold of Gem popularity below which there aren't enough eyeballs to spot problems and the Gem should be considered insecure? And if you do spot a problem, do you report it and hope the maintainer has time to do something about it? Or do you write a PR and submit it, hoping they accept it? Doesn't that then mean you're maintaining someone else's code base? Again, I would massively prefer to write and maintain my own code than maintain someone's else code (or wait for them to fix a problem that they may no longer care about). How do you build a secure application for something as trusted as Github while gleefully incorporating all this third-party code?

aantix 8y ago

Using the conditional boot loading, aren’t there structural differences in ActiveRecord queries/scopes that would run under 3.2 but not 5.2? Did GH just rewrite those scopes in their respective models and maintain a ton of if/else blocks for the different versions? And if so, didn’t they run into issues without the code not being DRY, e.g. someone fixes a 3.2 query, but not the corresponding 5.2 version?

barrkel 8y ago
If you have your test code unified, and have multiple CI pipelines, it should show up immediately on your build servers.

deleted 8y ago

[deleted]

stevebmark 8y ago

There are several "we upgraded Rails, it was huge, risky, and took months to years" blog posts from medium to large companies. I personally take this as a warning against using Rails. Ruby is one of the most dangerous dynamic languages to refactor, I don't see how struggling to do it for over a year is a selling point of the framework. It also feels counter to Rails's mantra of delivering value fast with little effort, until you need to upgrade, then you have months of no business value delivery and need to bring in experts to help.

majewsky 8y ago
> Ruby is one of the most dangerous dynamic languages to refactor A thousand times this. It's so much easier to do breaking changes and refactors in a language that's supporting you, instead of working against you.
- cknoxrun 8y ago
  Dynamic language or not, having good test coverage will be your biggest support.
  - weberc2 8y ago
    It's definitely important, but we (Python shop) have "good" (85%) test coverage and we still see 500s in prod every day because of things a type checker could trivially catch. And this is just in the course of normal operation; this isn't even a migration. Having extensive experience with both Go and Python, I would conservatively estimate that Go requires ~30% fewer tests than untyped Python for the same confidence. That's more than 30% time savings; not only are you not writing 30% of the tests, but that's 30% fewer tests to have to maintain. Of course these aren't the only considerations--for certain tasks Python may be faster to develop with (although I think people forget about things like deployment, tooling, dependency management, performance requirements, etc when they make their estimations).
    - jb3689 8y ago

tjpnz 8y ago

It's scarily common for organizations to be running ancient versions of Rails in production. At my last gig we spent six months upgrading a Rails 2.3 application to 3.2 and before that I was working with a team that was maintaining an application written in Rails 1x. Kudos to GitHub for sharing this, I really hope they do future posts going into more detail. In my experience one of the hardest aspects to upgrading Rails is that so much of the really useful information has either fallen out of Google or succumbed to link rot.

cptskippy 8y ago
It's not just Rails, this common with many frameworks and even turnkey solutions from vendors. Nothing will ever meet your needs 100% and so you end up with some customization. It really depends on how integral those customizations were and how tightly they are coupled to the product being used.
stouset 8y ago
This is true of literally everything. People don't upgrade their dependencies across the board, and it's a massive problem for long-term security and maintainability.
- jb3689 8y ago
  My current company doesn't even lock most of their dependencies. At first I thought it was crazy (and it is crazy) but it does mean we address compatibility issues immediately. We are mostly running background jobs though so it's safer than anything customer facing
  - stouset 8y ago
    My opinions on this have changed a lot over the past few years. At this point, I think anything that pins dependencies to specific versions is asking for long-term maintainability nightmares. Updates of your dependencies, operating system, language version, etc. should happen weekly, and any instance where you have to pin a dependency to an old version (e.g., a major version release that has some compatibility issues) should be dealt with ASAP. Obviously this isn't a tenable position in every circumstance, but I think it should be the default. Particularly in a world where the vast majority of security fixes go without an announcement or CVE.

throwaway427 8y ago

Given its maturity and settled place in the programming landscape it's always nice to see that Rails can still evoke irrational disdain in HN comments.

pwelch 8y ago
100%
gameswithgo 8y ago
Why is irrational to hate languages that are orders of magnitude slower than are necessary?
- jb3689 8y ago
  Something-something IO something-something
- goatlover 8y ago
  It's irrational to hate without mentioning tradeoffs. Sure, if performance is your only metric, then Ruby is a bad choice. But that's rarely the case, particularly with the web.
- rpeden 8y ago
  That's a reasonable question to ask. I think in general, there are lots of reasons to like a language outside of its runtime performance. I love working with Go and Rust due to their performance. Any I work every day in C#, which ends up nice and quick, too. But I still love Ruby due to its expressiveness, and the way it works just seems to align with the way I think. But that's poetically because I used Smalltalk in the past and I like the bits of it that Ruby borrowed. :) To answer to original question, though. I'd say it's irrational to hate languages that are slower than necessary because it's irrational to hate a programming language at all. No matter what language it is, it's just a bunch of words on a screen. Use the ones you like and don't waste any brain cycles thinking about the ones you don't. Unless you're locked in a cube farm and forced to write Cobol at gunpoint all day. Hate might be rational then.
- 8y ago

k__ 8y ago

I had the impression GH switched Ruby for Scala years ago.

marksomnian 8y ago
That was Twitter.
- k__ 8y ago
  Oh, lol. Thanks :)

Twirrim 8y ago

> Upgrade early and upgrade often It seems daft to keep seeing this lesson being learned by tech companies, and keep seeing blog posts where most of the pain would have been handled easily by just making upgrading a key feature. Instead, tech managers and engineers seem to make the same mistakes over and over again, delaying those upgrades, until suddenly they discover it's a hard task to upgrade. I get delaying to _some_ degree, it's better to let other people figure out those sharp bits on the bleeding edge for you, but you need to set an explicit target for upgrading. At another large tech company I worked for, it took the security team swinging the sledgehammer to get teams to upgrade from known-vulnerable versions of Ruby on Rails. When they came to do it, they discovered the changes were so extreme that the effort involved in migrating was likely more than the effort involved in a complete re-write (they did at least have pretty comprehensive tests)

cortesoft 8y ago
It is easy to say that in the abstract, but you always have a finite time/resource budget for doing work. Effort spent upgrading is effort not spent doing other important work. Is the other work more important in the long run? The answer is not trivial to answer. This is why we call it 'tech debt'.. it is just like any other debt. You take it on because you don't have the current resources to avoid it, and you calculate that it is worth taking it on. But then, you are carrying the interest on it, and if you aren't careful it will grow to be unmanageable, and all your dev effort goes into just paying the interest without paying the principle.
- Twirrim 8y ago
  It absolutely matters. Security is a feature. If you see upgrading as merely technical debt, you're never going to give it the appropriate attention.
  - regularfry 8y ago
    To twist the metaphor slightly, it's technical inflation.
  - 8y ago

gameswithgo 8y ago

How much money in server costs and how much electricity could be saved if Github didn't use an interpreted language, but something like Go, C#, F#, Java etc?

jb3689 8y ago
Github would not have been the same in any of those. They really took to some of the Rails concepts - a lot better than most Rails companies - and it shows in their product (routing, object structures, etc)
- gameswithgo 8y ago
  So your claim is you could not create the same user experience in any language that is jitted or compiled? I can't really take that seriously.
  - jb3689 8y ago
    Sure, in retrospect you could create the same thing. It's just various text processing at the end of the day. What I'm claiming is that their choice of Rails led to certain choices which were really transparent throughout the product and are still there. It would've grown to be something totally different on another technology, so I don't think it's fair to just look at cost and performance. They did many things "the Rails way"
erokar 8y ago
If they'd used Java they would still be working on the prototype.
- gameswithgo 8y ago
  I've seen quite a few attempts to measure productivity differences between different languages and there is not a consistent win being shown by dynamic languages in general. Perhaps ruby on rails is especially productive for the web, and maybe especially so when github.com launched, but there are lots of options now with similar productivity and 1 or 2 orders of magnitude better performance.

dwb 8y ago

Upgrading early(ish) and often, the very obvious preventative measure against terrible and failure-prone rewrite or upgrade projects, is one of the first things that falls by the wayside in the mostly short-termist logic that seems to dominate modern capitalism. It's absolutely infuriating.

conroy 8y ago

> The upgrade started out as kind of a hobby; engineers would work on it when they had free time. There was no dedicated team. I’m not sure why this still surprises me. For a company the size of Github, there should most certainly be a team responsible for these type of upgrades.

nautilus12 8y ago

Why in the world is github still on rails?

notriddle 8y ago
Why not?
stephenhuey 8y ago
Maybe because their codebase still serves their use cases very well? And perhaps they have little to gain and possibly much to lose if they ditch it? You didn't say much in your question, so I don't know if you feel they ought to rewrite with a popular SPA framework or use something like Elixir Phoenix, but if their Rails-based solution handily serves 30 million users, why do you feel so strongly they should move to something else?
- mrdoops 8y ago
  Nothing wrong with Rails, especially if the team knows it well. Time to develop is the real cost in software most of the time. If Github wanted to integrate a lot of real-time features, then Elixir + Phoenix can't be beat. Depending on what they replace, a 10x in performance and a fraction of the servers needed is a nice win.
  - innocentoldguy 8y ago
    I've seen performance boosts closer to 20x when I've helped companies rewrite their Rails products in Elixir. I've also seen a reduction in server costs. In all fairness though, simply rewriting the Rails app in Rails, with the benefit of hindsight, probably would have resulted in a performance gain too.
    - ksec 8y ago
      I would bet $10 the scaling problem with Github would properly have more to deal with Git than Rails itself. Switching to Elixir wouldn't really help.

config_yml 8y ago

What do they mean by off-hours? I imagine on a global site like github, there are hardly off-hours?

dyeje 8y ago
Just because it's a global site doesn't mean the traffic is distributed uniformly across the day. Certain regions are going to have higher traffic during business hours. I'd guess their off hours are somewhere around 6pm PST when North / South America has stopped working, Europe / Africa is asleep, and India is just waking up.
toasterlovin 8y ago
They probably mean when they weren't tied up shipping features or tracking down bugs.

jimnotgym 8y ago

The regular thread of people piling in to criticise dynamic languages. Instead perhaps people could suggest a better language/framework that is more productive than Rails, and has had a long lifespan in a large codebase?

BjoernKW 8y ago
Although arguably not as productive Java fares pretty well in that regard, particularly when it comes to maintainability. Still, Rails is terrific. If it wasn’t for Rails Java today wouldn’t be as productive either.
- alexhutcheson 8y ago
  Yeah Java isn't nearly as bad as it used to be, and it's pretty seamless to write your business logic in Kotlin if Java's verbosity annoys you.
  - AlphaSite 8y ago
    Honestly Java is getting better at a good pace now. It’s nowhere near as bad as the java 6 days.
radium3d 8y ago
I was surprised to learn that github runs on ruby on rails. Interesting, I didn't know Ruby has been around just as long as PHP. I still would choose PHP if my opinion matters, just from my experience with the slow performance of ruby on rails when I gave it a go just a few years back. PHP - 23 years Ruby - 23 years Java - 23 years JavaScript (nodejs) - 22 years (9 years) https://en.wikipedia.org/wiki/Dynamic_programming_language#Examples https://en.wikipedia.org/wiki/Dynamic_programming_language#E...
- bal00ns 8y ago
  Don't forget Python - 28 years

jbergstroem 8y ago

As a comparison, here's gitlab's journey (issue opened March, 2016): https://gitlab.com/gitlab-org/gitlab-ce/issues/14286 https://gitlab.com/gitlab-org/gitlab-ce/issues/14286 Looks like the first scheduled milestone was 9.5 (a year ago) and the current is set for 11.4 (next release).

lunaru 8y ago

For my team, this article comes at some interesting timing, since we're bumping into some of the same issues with Rails. Rails is now a mature framework and part of the problem is its lack of consideration for large existing codebases running in production. While there are nice tools to help migrate (e.g. rails:update) that hit surface issues, the deep problem is that there are a lot of decisions made going from version to version that are obviously unfriendly to established projects. e.g.: https://github.com/rails/rails/issues/27231 https://github.com/rails/rails/issues/27231 Additionally, there are a lot of gems that are losing momentum, which are near-core to Rails. e.g.: https://github.com/thiagopradi/octopus/issues/490 https://github.com/thiagopradi/octopus/issues/490. This is a side effect of the above issue, where the alternatives to Rails are taking a lot of the community away to focus on newer/shiner things. Fortunately, we have companies like GitHub and Shopify that are still very much invested in the success of the ecosystem. All that said, it's still a great framework to go from 0 to production with a new idea or project. Other ecosystems we're entrenched in (Node for example) have their share of issues as well, but we won't go into those.

lostapathy 8y ago
> art of the problem is its lack of consideration for large existing codebases running in production To be fair - this has gotten a lot better. Upgrading 2.3 -> 3.2 was terrible. 3.2 -> 4.0 was terrible. 4.0 -> 4.1 was rough. Since then, I've found the upgrades pretty easy - to the point I ran rails 5.2.0-rc's in production for a while. As you fairly note, a big problem is that related gems lose momentum and they don't get updated - which blocks other updates. On the flip side, they usually aren't that hard to update and submit a PR on, either. Even not having those patches merged quickly is not so bad in ruby - it's easy to tell bundler to look at your fork of a gem on github rather than pulling the upstream.
- bhaak 8y ago
  What was your problem with going from 3.2 to 4.0? I think the upgrade path from 2.3 being terrible is generally accepted as being true. But I don't remember any hard problems from 3.x onward. At least for Rails itself. Gems dependencies are another problem. Like at one project we had pains every time Rails upgraded their minor version because the previous devs thought using Squeel instead of the builtin ActiveRecord a good idea. Just for being able to write slightly "nicer" queries and now this is a major stopper for going to Rails 5.

ryenus 8y ago

Anyone knows which ruby runtime GitHub uses? Ruby MRI or JRuby etc.?

exabrial 8y ago

The advice at the end sounds exactly like something I'd say to someone going from 1.8 to 11 with Java. Great advice for any platform, very interesting to see the same conclusions from a totally different stack

starefossen 8y ago

As a person working for a large software consultancy in Scandinavia I hate to see so many using type safety as an excuse for not writing tests. At least a dynamic language forces you to write tests and frankly it is often easier to write tests in a dynamic language imho.

ksec 8y ago

I am very much looking forward to Rails 6.0 and see what Github / Shopify will upstream. Actually Instacart has lot of great gems too which I wish would have been the default solution in Rails.