7 ms·

GitHub: Scaling on Ruby, with a nomadic tech team

176 points by jssjr 11y ago

aaronbrethorst 11y ago
My favorite quote: this is actually a really pragmatic set of hackers that just hack on Ruby, hack on C and spend their time working on more interesting things using a more stable stack, rather than chasing after the latest and shiny tech.
- nailer 11y ago
  There's also the 'Ruby syntax and performance at the same time' option - which used to be limited to JRuby but is quickly being replaced with Elixir.
  - djur 11y ago
    Elixir's syntax is only superficially similar to Ruby's, and its semantics are radically different. I don't think there's any evidence that people who picked up JRuby for performance reasons are notably likely to leave Ruby for Elixir versus other languages.
- meesterdude 11y ago
  I liked this too. I definitely prefer working with a stack I know how to wield, and using it to solve problems. New things are good and useful, but having a focus means you can understand all the nuances that go along with it.
- bdcravens 11y ago
  Also worth noting that within their stack, they don't chase the newest versions. Until about a year ago, they were running Rails 2.3, when they upgraded to Rails 3.0: http://shayfrendt.com/posts/upgrading-github-to-rails-3-with-zero-downtime/ http://shayfrendt.com/posts/upgrading-github-to-rails-3-with...
grinnick 11y ago
How do you go from DB Admin to Director of Technology in 2 years!? That's pretty impressive.
- spectrum1234 11y ago
  One thought is the title Director is everywhere nowadays. Not to take anything away from this guy but it could be 2+ rungs below CTO. Probably just 1 though.
- samlambert 11y ago
  Thank you. It was a mix on working on impactful stuff https://github.com/blog/1880-making-mysql-better-at-github https://github.com/blog/1880-making-mysql-better-at-github, making an effort to work on more general issues like availability, having a culture that allowed me to be ambitious, and solid mentorship from our CTO.
otterley 11y ago
I'd love to see their compute and software stack provisioning tools. Are they open source?
- lswith 11y ago
  hubot is open source and you can write hubot scripts for things you want to do.
cdnsteve 11y ago
Very nice read. Simple, stable and steady wins platform battles. It seems to do well at allowing them to keep focus and careful control of their product. It's great to hear these core values are baked into their culture. I remember recently reading they were running Github pages on something like a pair of dedicated servers lol. Hats off, Github rocks. Our team recently started using Jira for issues instead of Github ones. I have to say, if Github offered a $1 more per repo cost for better issue/wiki tier, I'd rather use that in a second.
spectrum1234 11y ago
One of the best articles I've read in while
stijlist 11y ago
I'd be really curious to read more about how the Github team's intuition around polyglot programming and "Right Tool for the Job" rhetoric changed over time. I feel like I read a blog post about this one or two years ago but I can't remember which team member wrote it :)
- misiti3780 11y ago
  this one? https://github.com/blog/530-how-we-made-github-fast https://github.com/blog/530-how-we-made-github-fast
shiny 11y ago
> A month ago I was working from a cabin in the woods in Wisconsin. OT, but has anyone else tried this? I've often fantasized about working while embedded in a beautiful nature setting, but I imagine it's tough to find good internet in such a place. I'm in California ... any recommendations? I've also thought of copying Antony Garret Lisi's 'science hostel' idea [1] but for coders / entrepreneurs :) [1]: https://en.wikipedia.org/wiki/Antony_Garrett_Lisi#Science_hostel https://en.wikipedia.org/wiki/Antony_Garrett_Lisi#Science_ho...
- anon987 11y ago
  /r/digitalnomad
- sanswork 11y ago
  I bought an RV and did this for a while. It was fun but I wasn't any more productive. In the past I also tried moving to a remote beach resort town in Australia with a cofounder. It was fun and beautiful but again no more productive than we were back home.
  - snowwolf 11y ago
    Is that the point though? To be more productive? I'd be more interested to know if you were at least equally as productive and more importantly happier? I would think it's about improving your work/life balance and long term does that make you more fulfilled?
    - sanswork 11y ago
      Most people seem to think it will make them more productive without the distractions so for a lot of people it is. Pretty view won't improve your work/life balance either unfortunately. I was less productive and equally happy. All existences become the norm quickly. Even living in a tropical paradise, or surrounded by snow capped mountain wilderness.
- samlambert 11y ago
  I've been surprised how good the internet was out there actually. I'd totally recommend it. Waking up, going for a cycle ride through the woods before working on a porch in the sun is kind of amazing.
x404 11y ago
My favorite quote: "GitHub’s largely officeless workplace — about 60 percent of its employees work remotely, using a powerful homemade chatbot, called Hubot, to collaborate." It's really cool to work wherever you like without lose the team's efficient.
staunch 11y ago
> For a long time, very key bits of our infrastructure were strung together with Shell scripts and simple scripting, and it’s surprisingly effective and still works really very well for us. Key bits of the world's infrastructure still run on a bunch of flimsy shell scripts that seem like they'd break all the time but don't. If a computer program works reliably at all it will probably work reliably indefinitely, assuming the environment it lives in does not change substantially. We probably all trust our lives to high end routers and medical equipment powered by shell scripts (or worse) every day. The code would scare you but, surprisingly, it will probably never fail you.
- mbrock 11y ago
  It's interesting why shell scripts are seen as so "fragile" or "flimsy." Why would bash be any more inherently flimsy than any other interpreted language? Why would the global mass of Ruby, JavaScript, Perl, Python, Java, C++, whatever be any less fragile? The whole world is held together by duct tape. At least most shell scripts keep it simple.
  - danpalmer 11y ago
    > Why would bash be any more inherently flimsy than any other interpreted language? I'd say that it at least feels this way, because the "global state" in a shell program depends on a lot more properties of the system than the "global state" of a Python program. C/C++ or even entirely static binaries from Go for example (I'm not a Go expert) all eliminate state being 'imported' from the host machine.
    - mbrock 11y ago
      Sure, if the external tools used by the script change, the script's behavior will change... Other languages somewhat solve this with versioned dependencies, but even that's not quite sufficient, which is how come all the rage about reproducible builds. But yeah, good point.
  - anko 11y ago
    One of my pet peeves about shell scripts (speaking as a former UNIX admin) is the lack of static typing or test frameworks. So not only is the language less elegant than modern programming languages, but it's harder to test for bugs and you don't even get type safety.
mahyarm 11y ago
I wonder what they use for a chat system? Slack? IRC? Jabber? Their own app suite?
- Linell 11y ago
  IIRC, they use Campfire. https://campfirenow.com/ https://campfirenow.com/
why-el 11y ago
Another thing that would be interesting to learn more about is their Front end structure. From what I can gather they keep things extremely simple (jQuery, mostly) and make extensive use of pjax and server generated javascript that gets evaluated by the client. If this is all there is to it then it is really impressive, considering how many other companies of similar or smaller size quickly inject a Javascript MVC on top of their stack the minute the interaction goes past simple forms (Airbnb comes to mind, who to my knowledge is using Backbone for some aspects of their app). To be clear I think both approaches are excellent and all depends on the in-house talent, but nonetheless it's interesting to see how GitHub is sticking to a traditional, Rails-inspired document-based approach that minimizes front end complexity while other companies do not, even when both have similar UX complexity.

176 points by jssjr 11y ago

My favorite quote: this is actually a really pragmatic set of hackers that just hack on Ruby, hack on C and spend their time working on more interesting things using a more stable stack, rather than chasing after the latest and shiny tech.

nailer 11y ago
There's also the 'Ruby syntax and performance at the same time' option - which used to be limited to JRuby but is quickly being replaced with Elixir.
- djur 11y ago
  Elixir's syntax is only superficially similar to Ruby's, and its semantics are radically different. I don't think there's any evidence that people who picked up JRuby for performance reasons are notably likely to leave Ruby for Elixir versus other languages.
meesterdude 11y ago
I liked this too. I definitely prefer working with a stack I know how to wield, and using it to solve problems. New things are good and useful, but having a focus means you can understand all the nuances that go along with it.
bdcravens 11y ago
Also worth noting that within their stack, they don't chase the newest versions. Until about a year ago, they were running Rails 2.3, when they upgraded to Rails 3.0: http://shayfrendt.com/posts/upgrading-github-to-rails-3-with-zero-downtime/ http://shayfrendt.com/posts/upgrading-github-to-rails-3-with...

grinnick 11y ago

How do you go from DB Admin to Director of Technology in 2 years!? That's pretty impressive.

spectrum1234 11y ago
One thought is the title Director is everywhere nowadays. Not to take anything away from this guy but it could be 2+ rungs below CTO. Probably just 1 though.
samlambert 11y ago
Thank you. It was a mix on working on impactful stuff https://github.com/blog/1880-making-mysql-better-at-github https://github.com/blog/1880-making-mysql-better-at-github, making an effort to work on more general issues like availability, having a culture that allowed me to be ambitious, and solid mentorship from our CTO.

otterley 11y ago

I'd love to see their compute and software stack provisioning tools. Are they open source?

lswith 11y ago
hubot is open source and you can write hubot scripts for things you want to do.

cdnsteve 11y ago

Very nice read. Simple, stable and steady wins platform battles. It seems to do well at allowing them to keep focus and careful control of their product. It's great to hear these core values are baked into their culture. I remember recently reading they were running Github pages on something like a pair of dedicated servers lol. Hats off, Github rocks. Our team recently started using Jira for issues instead of Github ones. I have to say, if Github offered a $1 more per repo cost for better issue/wiki tier, I'd rather use that in a second.

spectrum1234 11y ago

One of the best articles I've read in while

stijlist 11y ago

I'd be really curious to read more about how the Github team's intuition around polyglot programming and "Right Tool for the Job" rhetoric changed over time. I feel like I read a blog post about this one or two years ago but I can't remember which team member wrote it :)

misiti3780 11y ago
this one? https://github.com/blog/530-how-we-made-github-fast https://github.com/blog/530-how-we-made-github-fast

shiny 11y ago

> A month ago I was working from a cabin in the woods in Wisconsin. OT, but has anyone else tried this? I've often fantasized about working while embedded in a beautiful nature setting, but I imagine it's tough to find good internet in such a place. I'm in California ... any recommendations? I've also thought of copying Antony Garret Lisi's 'science hostel' idea [1] but for coders / entrepreneurs :) [1]: https://en.wikipedia.org/wiki/Antony_Garrett_Lisi#Science_hostel https://en.wikipedia.org/wiki/Antony_Garrett_Lisi#Science_ho...

anon987 11y ago
/r/digitalnomad
sanswork 11y ago
I bought an RV and did this for a while. It was fun but I wasn't any more productive. In the past I also tried moving to a remote beach resort town in Australia with a cofounder. It was fun and beautiful but again no more productive than we were back home.
- snowwolf 11y ago
  Is that the point though? To be more productive? I'd be more interested to know if you were at least equally as productive and more importantly happier? I would think it's about improving your work/life balance and long term does that make you more fulfilled?
  - sanswork 11y ago
    Most people seem to think it will make them more productive without the distractions so for a lot of people it is. Pretty view won't improve your work/life balance either unfortunately. I was less productive and equally happy. All existences become the norm quickly. Even living in a tropical paradise, or surrounded by snow capped mountain wilderness.
samlambert 11y ago
I've been surprised how good the internet was out there actually. I'd totally recommend it. Waking up, going for a cycle ride through the woods before working on a porch in the sun is kind of amazing.

x404 11y ago

My favorite quote: "GitHub’s largely officeless workplace — about 60 percent of its employees work remotely, using a powerful homemade chatbot, called Hubot, to collaborate." It's really cool to work wherever you like without lose the team's efficient.

staunch 11y ago

> For a long time, very key bits of our infrastructure were strung together with Shell scripts and simple scripting, and it’s surprisingly effective and still works really very well for us. Key bits of the world's infrastructure still run on a bunch of flimsy shell scripts that seem like they'd break all the time but don't. If a computer program works reliably at all it will probably work reliably indefinitely, assuming the environment it lives in does not change substantially. We probably all trust our lives to high end routers and medical equipment powered by shell scripts (or worse) every day. The code would scare you but, surprisingly, it will probably never fail you.

mbrock 11y ago
It's interesting why shell scripts are seen as so "fragile" or "flimsy." Why would bash be any more inherently flimsy than any other interpreted language? Why would the global mass of Ruby, JavaScript, Perl, Python, Java, C++, whatever be any less fragile? The whole world is held together by duct tape. At least most shell scripts keep it simple.
- danpalmer 11y ago
  > Why would bash be any more inherently flimsy than any other interpreted language? I'd say that it at least feels this way, because the "global state" in a shell program depends on a lot more properties of the system than the "global state" of a Python program. C/C++ or even entirely static binaries from Go for example (I'm not a Go expert) all eliminate state being 'imported' from the host machine.
  - mbrock 11y ago
    Sure, if the external tools used by the script change, the script's behavior will change... Other languages somewhat solve this with versioned dependencies, but even that's not quite sufficient, which is how come all the rage about reproducible builds. But yeah, good point.
- anko 11y ago
  One of my pet peeves about shell scripts (speaking as a former UNIX admin) is the lack of static typing or test frameworks. So not only is the language less elegant than modern programming languages, but it's harder to test for bugs and you don't even get type safety.

mahyarm 11y ago

I wonder what they use for a chat system? Slack? IRC? Jabber? Their own app suite?

Linell 11y ago
IIRC, they use Campfire. https://campfirenow.com/ https://campfirenow.com/

why-el 11y ago

Another thing that would be interesting to learn more about is their Front end structure. From what I can gather they keep things extremely simple (jQuery, mostly) and make extensive use of pjax and server generated javascript that gets evaluated by the client. If this is all there is to it then it is really impressive, considering how many other companies of similar or smaller size quickly inject a Javascript MVC on top of their stack the minute the interaction goes past simple forms (Airbnb comes to mind, who to my knowledge is using Backbone for some aspects of their app). To be clear I think both approaches are excellent and all depends on the in-house talent, but nonetheless it's interesting to see how GitHub is sticking to a traditional, Rails-inspired document-based approach that minimizes front end complexity while other companies do not, even when both have similar UX complexity.