8 ms·

I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Ou

by cbb330 3y ago

I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Our monolith flagship web app. And, there’s more systems at LinkedIn with MM lines of code and long builds example is the mid tiers, the offline data stacks, the metrics systems, KafkaKafkaKafka. Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good. Anyways, AMA

hasty_pudding 3y ago
I worked at LinkedIn in infra. Their internal tools are a nightmare. The true definition of Jugaad. The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing. I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.
- cbb330 3y ago
  TIL jugaad haha. Yea, give everything 3x estimate because everything that can break will break along the way. Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.
- makingstuffs 3y ago
  Seeing you use 'Jugaad' make me unreasonably happy. I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution. Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.
  - rrr_oh_man 3y ago
    But are things ever permanent? Isn’t it to be expected that stuff can change in the future?
    - hasty_pudding 3y ago
      LinkedIn thinks so. Their tech stack is so over engineered and spaghetti that they spent a hundred million and years trying to move into Azure and were beaten by how bloated and unmovable the tech stack is. Theyre still using python 2 and centOS in tons of systems. They just started using github last year. Their tech stack is easily over a decade old.
      - Domenic_S 3y ago
        > Theyre still using python 2 and centOS in tons of systems. There's 1 service on py2, with a plan to migrate because of the move off of centos. centos deprecation is in progress.
        hasty_pudding 3y ago
        > There's 1 service on py2 Are you sure about that? CentOs migration has been in progress for years lol
- dustingetz 3y ago
  jugaad: https://en.m.wikipedia.org/wiki/Jugaad https://en.m.wikipedia.org/wiki/Jugaad
- notso411 3y ago
  [dead]
- yellow_lead 3y ago
  In my first SWE job, I was perplexed, I never felt much impact for solving complex bugs, even if they had side effects like reducing latency for everyone in our infra. Anyways, I realized it's like you say - you get rewarded for pushing features and shipping stuff. This incentive can easily become a 'tragedy of the commons' situation.
pm90 3y ago
Why did the codebase end up like this? Is it a lack of hiring for platform or developer tooling teams?
- z3dd 3y ago
  I bet in most places it all starts with a new innovative system of (self) assessment that relies on number of features/VI/KPI tied to promotions.
- selfmodruntime 3y ago
  It's because LinkedIn is old in terms of web-age. Really old. They, along with thousands of other companies, had the need for a reactive web front-end when JavaScript had not yet matured to where it is now.
  - cbb330 3y ago
    this plus lack of performance eval/culture/strategy to grow responsibly and enable future success its a great learning experience though despite what people say about the inability to learn the new hot hot tech. the nuance of software development is the decisions that other people make, that is inescapable and is a skill worth developing. i'm not buying the "just do a startup" because i think its a cop out.
- lvspiff 3y ago
  https://xkcd.com/927/ https://xkcd.com/927/
- marcinzm 3y ago
  In my experience, a good platform team will increase velocity a bit while a bad platform team will tank it. Perpetual migrations and re-writes, "simplified" platforms that fail to meet future developer needs, half-abandoned NIH systems as people move teams, ownership moats if product teams try to touch platform code, etc, etc. So if a company has been burned a couple times they may decide to just not bother.
pcloadletter_ 3y ago
This sounds like my exact experience at another big tech company
Aeolun 3y ago
> Unfortunately, 17min to build is pretty good. My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that. As opposed to making builds faster, or build infra faster/cheaper.
- notso411 3y ago
  [dead]
- zaphar 3y ago
  There are mostly only a few ways to make builds faster. * Ship less code (Very hard to get a large org to do) * Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code. * Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building. These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.
  - Aeolun 3y ago
    These kind of speedups at LinkedIn scale all fall off the chart at the bottom left of this scale: https://xkcd.com/1205/ https://xkcd.com/1205/ Nonetheless, leadership will never give it priority.
  - recroad 3y ago
    Can you expand on "ship less code"? I think I'm agreeing with you but I'm taking it to mean ship smaller pieces of code, not overall LoC.
    - zaphar 3y ago
      I mean both. Smaller pieces but also less of those pieces. It is my experience that many systems code size is less purely a function of what they need to do than a complex set of layered components and systems in the name of ease and fear of NIH syndrome. Sometimes doing the same thing with simpler components and fewer components is just better.
AndrewKemendo 3y ago
>Anyways, AMA What precisely is LinkedIn trying to be? Seems like it's turning into 2007 Facebook. Is that intentional?
- duxup 3y ago
  I wonder, is "being facebook" is that an organic goal LinkedIn came up with, or did they pick that up from their users? I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.
  - bombcar 3y ago
    I've always thought LinkedIn was just Facebook for Business™ - has it ever been anything else?
    - duxup 3y ago
      It was always to some extent, but the feed was a lot more formal IMO at the beginning. Content was more like actual articles and links. But I think users just love / gravitate towards a more facebook style feed.
      - bombcar 3y ago
        Part of that might be the "reward for spamming" - any "social media" site is always amazing at the beginning, because only the die-hards know about it and the spammers have no return on investment. LinkedIn has slightly different style of spammer, but it's basically the end game for anything.
  - disgruntledphd2 3y ago
    Nah, basically every other social type thing got screwed by FB convincing everyone that MAU's (i.e. the metric FB looked best on) was the right metric, when it clearly wasn't. Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).
yellow_lead 3y ago
Whats part two of the story? Did LinkedIn move to React? Did the initiative fail, or is it ongoing?
- cbb330 3y ago
  Preface: I’m not in front end or flagship teams so I’m saying hearsay. The initiative Chris refers to is still alive but likely hasn’t made any meaningful progress. These things tend to have a lot of fan fair and then suddenly leave the front conscious of the company as we get a new GDPR/DMA fire to put out. at that point it will be dead. Or, it will stay alive for 2+ years as slowly but surely 50% of clients migrate to it and see even poorer performance. There was a blog here a few months ago about migration from restli to grpc. It’s still going on, and nearly every app i see is still restli.
  - throwaway2day03 3y ago
    dude, you are sharing way too much about us publicly. the status of restli -> grpc and talking about DMA? seriously? you realize stylometry exists, right? even the phrase 'fan fair' is a strong signal.
    - smsx 3y ago
      Sounded like he he was talking about a public blog post right?
      - cbb330 3y ago
        https://www.infoq.com/news/2023/07/linkedin-protocol-buffers-restli/ https://www.infoq.com/news/2023/07/linkedin-protocol-buffers... https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/europe-fit-digital-age/digital-markets-act-ensuring-fair-and-open-digital-markets_en https://commission.europa.eu/strategy-and-policy/priorities-... https://gdpr-info.eu/ https://gdpr-info.eu/
  - yellow_lead 3y ago
    That's understandable, thanks for the info!
MisterPea 3y ago
Just for another perspective, I've worked at a few companies including LinkedIn as a backend developer and I think LinkedIn is probably around 70-80th percentile in code quality. There was significant emphasis on code quality at least on the team I was on, and an ever improving culture. I did work on one voyager task though and I remember it being a nightmare
- deleted 3y ago
  [deleted]

by cbb330 3y ago

hasty_pudding 3y ago

I worked at LinkedIn in infra. Their internal tools are a nightmare. The true definition of Jugaad. The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing. I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.

cbb330 3y ago
TIL jugaad haha. Yea, give everything 3x estimate because everything that can break will break along the way. Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.
makingstuffs 3y ago
Seeing you use 'Jugaad' make me unreasonably happy. I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution. Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.
- rrr_oh_man 3y ago
  But are things ever permanent? Isn’t it to be expected that stuff can change in the future?
  - hasty_pudding 3y ago
    LinkedIn thinks so. Their tech stack is so over engineered and spaghetti that they spent a hundred million and years trying to move into Azure and were beaten by how bloated and unmovable the tech stack is. Theyre still using python 2 and centOS in tons of systems. They just started using github last year. Their tech stack is easily over a decade old.
    - Domenic_S 3y ago
      > Theyre still using python 2 and centOS in tons of systems. There's 1 service on py2, with a plan to migrate because of the move off of centos. centos deprecation is in progress.
      - hasty_pudding 3y ago
        > There's 1 service on py2 Are you sure about that? CentOs migration has been in progress for years lol
dustingetz 3y ago
jugaad: https://en.m.wikipedia.org/wiki/Jugaad https://en.m.wikipedia.org/wiki/Jugaad
notso411 3y ago
[dead]
yellow_lead 3y ago
In my first SWE job, I was perplexed, I never felt much impact for solving complex bugs, even if they had side effects like reducing latency for everyone in our infra. Anyways, I realized it's like you say - you get rewarded for pushing features and shipping stuff. This incentive can easily become a 'tragedy of the commons' situation.

pm90 3y ago

Why did the codebase end up like this? Is it a lack of hiring for platform or developer tooling teams?

z3dd 3y ago
I bet in most places it all starts with a new innovative system of (self) assessment that relies on number of features/VI/KPI tied to promotions.
selfmodruntime 3y ago
It's because LinkedIn is old in terms of web-age. Really old. They, along with thousands of other companies, had the need for a reactive web front-end when JavaScript had not yet matured to where it is now.
- cbb330 3y ago
  this plus lack of performance eval/culture/strategy to grow responsibly and enable future success its a great learning experience though despite what people say about the inability to learn the new hot hot tech. the nuance of software development is the decisions that other people make, that is inescapable and is a skill worth developing. i'm not buying the "just do a startup" because i think its a cop out.
lvspiff 3y ago
https://xkcd.com/927/ https://xkcd.com/927/
marcinzm 3y ago
In my experience, a good platform team will increase velocity a bit while a bad platform team will tank it. Perpetual migrations and re-writes, "simplified" platforms that fail to meet future developer needs, half-abandoned NIH systems as people move teams, ownership moats if product teams try to touch platform code, etc, etc. So if a company has been burned a couple times they may decide to just not bother.

pcloadletter_ 3y ago

This sounds like my exact experience at another big tech company

Aeolun 3y ago

> Unfortunately, 17min to build is pretty good. My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that. As opposed to making builds faster, or build infra faster/cheaper.

notso411 3y ago
[dead]
zaphar 3y ago
There are mostly only a few ways to make builds faster. * Ship less code (Very hard to get a large org to do) * Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code. * Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building. These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.
- Aeolun 3y ago
  These kind of speedups at LinkedIn scale all fall off the chart at the bottom left of this scale: https://xkcd.com/1205/ https://xkcd.com/1205/ Nonetheless, leadership will never give it priority.
- recroad 3y ago
  Can you expand on "ship less code"? I think I'm agreeing with you but I'm taking it to mean ship smaller pieces of code, not overall LoC.
  - zaphar 3y ago
    I mean both. Smaller pieces but also less of those pieces. It is my experience that many systems code size is less purely a function of what they need to do than a complex set of layered components and systems in the name of ease and fear of NIH syndrome. Sometimes doing the same thing with simpler components and fewer components is just better.

AndrewKemendo 3y ago

>Anyways, AMA What precisely is LinkedIn trying to be? Seems like it's turning into 2007 Facebook. Is that intentional?

duxup 3y ago
I wonder, is "being facebook" is that an organic goal LinkedIn came up with, or did they pick that up from their users? I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.
- bombcar 3y ago
  I've always thought LinkedIn was just Facebook for Business™ - has it ever been anything else?
  - duxup 3y ago
    It was always to some extent, but the feed was a lot more formal IMO at the beginning. Content was more like actual articles and links. But I think users just love / gravitate towards a more facebook style feed.
    - bombcar 3y ago
      Part of that might be the "reward for spamming" - any "social media" site is always amazing at the beginning, because only the die-hards know about it and the spammers have no return on investment. LinkedIn has slightly different style of spammer, but it's basically the end game for anything.
- disgruntledphd2 3y ago
  Nah, basically every other social type thing got screwed by FB convincing everyone that MAU's (i.e. the metric FB looked best on) was the right metric, when it clearly wasn't. Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).

yellow_lead 3y ago

Whats part two of the story? Did LinkedIn move to React? Did the initiative fail, or is it ongoing?

cbb330 3y ago
Preface: I’m not in front end or flagship teams so I’m saying hearsay. The initiative Chris refers to is still alive but likely hasn’t made any meaningful progress. These things tend to have a lot of fan fair and then suddenly leave the front conscious of the company as we get a new GDPR/DMA fire to put out. at that point it will be dead. Or, it will stay alive for 2+ years as slowly but surely 50% of clients migrate to it and see even poorer performance. There was a blog here a few months ago about migration from restli to grpc. It’s still going on, and nearly every app i see is still restli.
- throwaway2day03 3y ago
  dude, you are sharing way too much about us publicly. the status of restli -> grpc and talking about DMA? seriously? you realize stylometry exists, right? even the phrase 'fan fair' is a strong signal.
  - smsx 3y ago
    Sounded like he he was talking about a public blog post right?
    - cbb330 3y ago
      https://www.infoq.com/news/2023/07/linkedin-protocol-buffers-restli/ https://www.infoq.com/news/2023/07/linkedin-protocol-buffers... https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/europe-fit-digital-age/digital-markets-act-ensuring-fair-and-open-digital-markets_en https://commission.europa.eu/strategy-and-policy/priorities-... https://gdpr-info.eu/ https://gdpr-info.eu/
- yellow_lead 3y ago
  That's understandable, thanks for the info!

MisterPea 3y ago

Just for another perspective, I've worked at a few companies including LinkedIn as a backend developer and I think LinkedIn is probably around 70-80th percentile in code quality. There was significant emphasis on code quality at least on the team I was on, and an ever improving culture. I did work on one voyager task though and I remember it being a nightmare

deleted 3y ago
[deleted]