8 ms·

You can't git clone a team

115 points by plam503711 1y ago

polishdude20 1y ago
I think the issue started way before AI when it comes to not having enough fresh blood for this industry. Companies have decided they don't want to train juniors. They want seniors who already have the skills.
- somanyphotons 1y ago
  This just means the market is saturated. When there aren't enough people they'll accept less experience
  - betterThanTexas 1y ago
    ...this doesn't make sense. Surely you need to factor in price point. Often times junior engineers deliver disproportionate value. Some ratio of juniors:seniors just seems rational, and those juniors grow into seniors. Maybe there's a good argument against training, but it could also just be irrational and stubborn in this case.
    - gbuk2013 1y ago
      In my experience it only works if you pull in juniors into strong teams and keep the proportion of juniors reasonably small. You also need to have a process in place for training them - it’s not enough to rely on ad-hoc mentoring from peers. If you get the ratio wrong, with too many juniors and weak technical leadership then you will end up in a very bad place in your code base. In terms of value, even if juniors are half the cost, it is much wiser to hire one senior instead of 2 juniors for the same money.
      - betterThanTexas 1y ago
        > it is much wiser to hire one senior instead of 2 juniors for the same money. Maybe in terms of pure productivity, but if you can match hiring to roadmaps you can give them more approachable/further from revenue work for which seniors would be an overkill. Etc. I'm just saying anyone with a simple explanation is only telling you part of the story. Besides, hiring is expensive. If you say live near a university, you have an edge in finding talent.
        gbuk2013 1y ago
        > give them more approachable/further from revenue work for which seniors would be an overkill My experience suggests there is no such thing. There is work that seniors might not want to do, but if you hired well then they will be professional enough to do it, if it really needs to be done. And it takes some experience to determine which “non-revenue generating” work (e.g. tech debt) actually needs to be done, to advocate doing it to the stakeholders and to actually do it well. Juniors need a lot of supervision and that is not free. Which is not a reason not hire them in the first place, just that that it should be done mindfully.
        johnnyanmac 1y ago
        >which “non-revenue generating” work (e.g. tech debt) actually needs to be done This mentality is part of the problem. You can't fundamentally treat juniors as a profit center. Especially in more niche fields. They need to be trained and schools don't cover your specific pipeline. maybe it doesn't "need" to be done, but some refactoring work will pay off to turn your juniors today into seniors tomorrow. If you can only think in productivity charts, then you don't hire juniors.
  - jandrewrogers 1y ago
    This doesn’t follow. Below some level of skill and experience they can contribute negative value to the project. Companies need a minimum level of experience just to make the role pay for itself. Does the high skill standard for surgeons mean the market for surgeons is saturated?
    - 9rx 1y ago
      If you are not needing to consider having someone who does not meet a certain skill standard to perform surgery on you, then, yes, it would seem that the surgeon market is saturated as the parent describes it.
      - phito 1y ago
        It depends on the time scale you are looking at it with. It looks saturated short-term, but it surely is not long term. The problem will only show up once your surgeons start retiring.
        9rx 1y ago
        Instantaneous time is the only time scale that makes sense with respect to the topic at hand. Sure, it is possible for the market to desaturate at some point as key people leave. But at that point, assuming suitable, per the earlier comment, replacements don't fill the void, the market will either come to accept lesser skilled people or it will come to accept fewer surgeries, returning to equilibrium. Not exactly magic.
- n_ary 1y ago
  > Companies have decided they don't want to train juniors. They want seniors who already have the skills. It got worse in last two years. Now Senior Engineers must have exact combo of the weird tech stack and tools with N years of experience exactly as the existing employees, else gtfo, you don't even get screening calls. You don't have something from nice to have list? Lol, why are you wasting our recruiter's time even? She needs to use GenAI to write her next rejection email. Also, your 15yoe does not matter, unless you are coming from direct competitor, in which case your 1.5yoe with internship is also excellent.
- gbuk2013 1y ago
  I’m a lead consultant on a gig where the CTO outright told us that he’s not really interested in growing their (frankly, under qualified) engineers, just wants them to get on with the job - it’s very short-sighted and sad. That said, in my previous job in a startup we hired very junior engineers and gave them plenty of opportunity and support for growth and several people did stunningly well. Pity the company didn’t do very well (IMHO due to focusing more or this sort of thing rather than making money). The truth is bound to be somewhere in the middle.
  - htrp 1y ago
    >That said, in my previous job in a startup we hired very junior engineers and gave them plenty of opportunity and support for growth and several people did stunningly well. Pity the company didn’t do very well (IMHO due to focusing more or this sort of thing rather than making money). Mentoring is a key part of technical leadership, way easier to help the talent grow into your requirements (even if they are under-qualified, for now)
    - plam503711 1y ago
      I agree but there's also the extra difficulty to do mentoring remotely (we are a remote first company). I *really* like being remote first and provide the choice if you want to work on site or wherever you want. But it does come with some challenges.
      - johnnyanmac 1y ago
        This may be a hot take, but I don't think most juniors should be remote. You need some time face to face to understand the processes and you miss lot of passive knowledge when you're not navigating a workplace of people discussing matters. once you get a few years under the belt, sure. You're probably fine doing everything online. I can't think of many remote only studios hiring juniors anyways.
        plam503711 1y ago
- artyom 1y ago
  I agree but it was always like that. Also there isn't any incentive anymore in growing juniors if they're going to immediately take the next better offer that comes around when they know enough, and you have such thing being offered on a daily basis (at least until a couple years ago). I would love to work on low-level, systems stuff (anything as close to the hardware as possible), that's even my education and area of expertise. BUT SaaS companies in reality: - pay better - have lower costs - are way easier - don't have that many (or any at all) geographical restrictions (e.g. importing hardware prototypes). People follows the money.
  - johnnyanmac 1y ago
    >there isn't any incentive anymore in growing juniors if they're going to immediately take the next better offer that comes around when they know enough okay. Promote them with proper pay then. This HR debacle of putting more budget into hiring over retaining is entierly self-inflicted. >People follows the money. Well that love sounds more like a low priority whimsy in that case. I don't think tech workers of all people are ones who ever complain about low compensation. Unless you work in games, I suppose.
- jandrewrogers 1y ago
  I think it is important to acknowledge that it takes much longer to train juniors in systems software today than a couple decades ago because the nature of the problems have changed. It now takes several years to get someone to an acceptable level for a lot of systems software work. In many cases that’s longer than the entire development cycle — a junior would never really be productive on the project. And yes, this is creating a vicious feedback loop where we are no longer producing many new people with these skills despite a lot of demand. The minimum level of sophistication required to be effective in systems software has increased dramatically since the 1990s, when I first started doing systems software. The kinds of systems software we were putting in production in back then would be considered a trivial toy today. This shift has placed an enormous amount of upward pressure on the minimum level of experience and skill that would allow someone to become productive within a useful amount of time. It is no longer feasible in many cases to have companies effectively subsidize the development of highly skilled systems software people. The training time has become so long that companies will never recoup the investment. It is easy to see how the incentives have created the current situation and it is not clear how to address the shortage. Even before the current situation, it was widely noted that most systems software developers were self-taught over many years rather than trained up in a structured environment.
  - johnnyanmac 1y ago
    >And yes, this is creating a vicious feedback loop where we are no longer producing many new people with these skills despite a lot of demand. if the company would rather burn out and age out all their talent than invest in the future, I don't know what to say. For niche domains, you can't treat talent a a pure business expense. That's how you lose out all that talent to China and suddenly you can't compete at all. If anything, training should be treated like insurance. >It is no longer feasible in many cases to have companies effectively subsidize the development of highly skilled systems software people. "subsidize" implies that somehow engineers are saving money by choosing to take these more niche paths. It doesn't make sense. You have a product and you need talent to develop and maintain it to compete. There's no "subsidies" in a business relationship like this. your choices are few and simple: train in-house, lobby schools to train for you, or publicize your tech and hope people train themselves for you again. It seems like these days companies aren't looking out long term though and just want to jump the ship that has plenty of time to steer past the iceberg everyone sees ahead.
tra3 1y ago
Maybe I'm being too cynical, but I can't tell whether this is a promotional piece for something.. That said, I do agree with the premise of the article that it's hard to learn "the stack", especially with the advent of generative AI. "Back in the day", when google spat out a link to something resembling your problem, you still had to deconstruct the answer given to apply it to your particular case. That required some sort of understanding, or a mental model of the domain. With ChatGPT I find myself copying/pasting error messages between two browser windows, or a terminal, not really understanding what is going on. No learning is happening. Does not bode well for new folks coming in.
- dingnuts 1y ago
  doesn't bode well for you either
  - tra3 1y ago
    "A problem well stated is a problem half solved", is all I got for now.
- plam503711 1y ago
  Hi! It's not a promotional piece of something, it's my personal experience as a CEO and co-founder of a company using Xen as the core of our stack. I like to share my views in a transparent fashion on how it's hard to do very technical stuff, not just for technical reasons, but due to the lack of people trained for.
- maximumgeek 1y ago
  If you find that to be the case when you use ChatGPT, maybe you should quit using ChatGPT. Sometimes, probably most of the time, it is better to work through it and understand an issue than to blindly copy pasta.
- pier25 1y ago
  > No learning is happening. Does not bode well for new folks coming in. Recently got into a completely new language/framework and used Copilot to understand what was going on. I still made all the big decisions and wrote most of the code myself. You can definitely use AI to avoid learning though (up to a point).
throw093209 1y ago
> System teachers: now rarer than RGB-free laptops LLM are quite good at explaining systems and frameworks. I would never got into kernel programming without Deepseek guidance. As for universities: too expensive, too much paperwork, too slow, too elitist.
- plam503711 1y ago
  It's true but until some extent. When you are talking about a hypervisor (like Xen), and many many subtle things depending on your CPU brand/model, it's really really *hard*, even with an LLM (and even more with an LLM hallucinating some CPU features or forgetting basic things like Meltdown and Spectre). However I agree: to learn a topic, LLMs are providing a great speedup. As a CEO/co-founder, I have no issue to hire people without a degree if they are good at what they do. However, our biggest chances are to scout directly in universities to find motivated students (motivation >>> everything else)
  - throw83394848 1y ago
    As a CEO, are you going to hire that motivated student? Or is it some sort of unpaid internship?
    - plam503711 1y ago
      We do not have any unpaid internship. So yes, I will hire them, until we reach a limit of mentors for them, because we cannot leave them alone.
citizenpaul 1y ago
Somebody tell that to the MBA's.
thenewwazoo 1y ago
As an aside, TFA says: We’re talking about skills that span kernel-level programming, hardware quirks, low-level debugging, distributed systems, security, orchestration logic, even the capability to work with the UI/UX team... and the ability to explain all that without scaring interns. You can’t just hire for that. You have to grow it. Nurture it. Beg for it. Or in some cases, resurrect it. If you are that person, what is the best way to market yourself? I am the person described. I've got experience from poking registers in firmware, to wireline transport protocol implementation, to infosec, to writing microservice framework middleware, to pipeline orchestration at the OS level, and on and on. In the last week I've debugged Linux UDS issues and TLS cipher suite problems, and wrote code to provision WiFi-connected devices over BLE. But it's incredibly hard to demonstrate that in an interview, if I can even find a role that warrants it. You're not going to find me on a university campus or in a research lab because I'm at a FAANG trying to pay my mortgage.
- frainfreeze 1y ago
  I found it is best to keep quiet, not even have a (directly attached to your name) blog with such varried content, and instead just send appropriate version of your CV when needed. When it comes to interviews I found it helps that I'm in my office/lab space, able to pick up the cam and show the person on call contents of my shelves and present few details about them on the office whiteboard. But even then they of course doubt you. Tldr raise your price and they'll belive easier.
  - ziddoap 1y ago
    The question was "what is the best way to market yourself?" and your recommendation is to keep quiet and do less marketing. If that has worked for you, that's amazing! But it seems really counter-intuitive to me.
    - zeta0134 1y ago
      I think the counterintuitive thing is that the marketing communicates a lack of confidence. It's... sortof odd to think about it that way, but to communicate actual confidence in your field, you mostly have to be willing and able to have a conversation about whatever topic your interviewer fancies. Being able to do that comfortably speaks volumes that your resume and project portfolio struggle to replicate.
neilv 1y ago
> What we’re doing about it (before it’s too late) I don't see "pay like Google" listed.
- plam503711 1y ago
  Because we cannot afford that, we aren't Google, and still a relatively small company vs the task of building a full-stack virtualization solution. Luckily, we have other strong points helping a lot (remote first, no micro-management, a great culture promoting human values and so on etc.)
koala_man 1y ago
The article starts off saying that if you want people with real full stack experience, from kernel to UX, you need to grow it. It goes on to say that it's hard to find and develop expertise for low level software like hypervisors. What's the connection between the topics? It feels like two different rants. If it's difficult to find kernel developers then wouldn't it help to not require them to also know web UX?
- ajsnigrutin 1y ago
  > If it's difficult to find kernel developers then wouldn't it help to not require them to also know web UX? That means hiring two people, and in $current_year, companies expect one person to know everything. Sysadmin, backend programmer, frontend programmer, designer and a DBA used to be different people not that long ago, now they expect one person to do all that... + it seems they want kernel development experience now.
  - delusional 1y ago
    Before they were multiple people they were one person. A single person can in fact write a program for a computer.
    - ajsnigrutin 1y ago
      Sure, some C code, some html, a table here, a colspan there, and you can have a website made by a single person... if we want a website to look like it was made on an 1980s computer by a single person.
      - wizzwizz4 1y ago
        Or, you can have a decent website made by a single person. It's not that hard to learn basic HTML5, enough ARIA semantics to know not to use ARIA, a programming language with decent synchronisation primitives that supports CGI, an SQL dialect and the principles of relational database design, enough JavaScript to use MDN, enough CSS to use MDN, the basics of server maintenance, TLS certificate provision, and DNS. If you want to do your own networking, or run email, that's a whole 'nother specialism; but just running a website is easy enough for one person to do.
smileysteve 1y ago
The title holds for much more specific and thinner stacks, along with support, QA, and sales - especially even simple web apps. Onboarding is important Training is important Retaining is important Maybe your system is 100% documented, conventional (looking at you Rails, Angular), debt free, tested, instrumented - but more than likely it's not. But if you get down to staff who can't teach a a system, including product teams that don't respect feature overload, value, or internal feature training. If you prioritize new features over team development and training (aka a team that doesn't know the system), you're likely to get muddy with existing features both technically and use facing.
- plam503711 1y ago
  I agree. If it's already hard for simpler stacks, you can imagine how hard it is for more critical or complex ones. And it's even worse if you inherited some of it (ie collecting technical debt that's not yours, which is the case here as some part of the stack are the result of a fork). Sometimes it's even a catch-22 situation, where the technical/generic knowledge is already hard to find, but you absolutely need it to train more junior people. Luckily we found such experimented people, but then you also need to use their expertise to actually fix stuff and not just mentor juniors. A very very delicate balance to find, especially in a timed market.
- mystified5016 1y ago
  My boss is about to learn this lesson the hard and painful way. Retention has fallen to the point where I am the only person who groks the full stack. I'm making an attempt to document what I can, but they're going to be fucked when I leave. I would feel bad about it, but CEO has tried to fuck me over many times-- for no apparent reason.
lesser-shadow 1y ago
>Partnering with Universities and Research Labs Like where? In the US? Western Europe? You're cleary not searching hard enough because I've never heard about your company once throughout my time at university.
- johnnyanmac 1y ago
  They shoutout a city in France, so I assume western EU.
deleted 1y ago
[deleted]
gitroom 1y ago
been there with bosses not caring about team learning at all, kinda messes everything up long run tbh
lammalamma25 1y ago
Curious if there is a way to dip your toe into this kind of thing without dedicating years of learning. Maybe a home lab? I've always been interested in this because it seems hard/interesting, and enjoyed learning about it in school. Wondering how much time you'd have to invest to see if you like it. I assume there is a healthy amount of "this is hard/not fun" along the way before you find out.
cadamsdotcom 1y ago
The value of knowledge is plummeting. It can, will, and should be subsumed by LLMs. Consider London’s black cabs. Drivers pass a gruelling exam to prove they know “everything” about one part of London. But, Google Maps puts that knowledge and more on every rideshare driver’s dashboard, for free, no advance study needed. Map knowledge doesn’t make you able to drive. Driving happens in real time - steering, obeying road rules, avoiding accidents. In engineering, the real time part is when you use mental models you’ve painstakingly developed - how the hardware works so you can debug it, what syntax is valid in your programming languages, what APIs exist in your stack. But, those depend on knowledge - so mental models can be self-taught with AI as tutor in your own time. The LLMs supply the knowledge, you integrate it via study - or a side project. An organization cannot clone itself a team, it’s true. They need engineers now, not in 6-12 months. But - a motivated individual can use AI to make themselves employable as a full stack engineer faster now than at any point in history. How cool is that!
- jandrewrogers 1y ago
  One of the big issues with LLMs for e.g. systems software is that there are broad domains where training data effectively does not exist. Consequently, you can’t learn much from them of value. It is the blind leading the blind. The lack of training material for LLMs is of course a lack of training material for people too. Some areas of software have a long history of relying almost entirely on an oral tradition to pass down knowledge. This has some advantages but it doesn’t scale and it makes it basically “dark knowledge” for LLMs or people without access to those that know it. If you want to get into an area like this, you often need to find a way to spend a lot of time with people that already know it.
  - johnnyanmac 1y ago
    >This has some advantages but it doesn’t scale and it makes it basically “dark knowledge” for LLMs or people without access to those that know it. We call this "tribal knowledge" in games. I despise it. It's done for a few reasons: - NDAs make public knowledge a landmine. And every game studio makes you sign NDAs. Even at the interview stage. - Churn. No one gets time to really develop expertise as they work on a project for 2-3 years and then layoffs come. Only a relative few become experts, and probably not because of the studio itself - lack of incentives. Games aren't very connected with acedemia to begin with, despite relying so much on cutting edge tech. So the best resources for sharing such techniques is shafted. This is slowly getting better as more tech conferences talk about games tech, but it's a pretty slow tricke unless you come from one of the largest studios and specifically come in for R&D. >If you want to get into an area like this, you often need to find a way to spend a lot of time with people that already know it. All too true. Open source development is one bastion for this, but that's overall why I keep trying to stay in this domain. You literally can't get the knowledge elsewhere. And it's knowledge that directly leads to better looking, more optimal, and less buggy games overall.
ygouzerh 1y ago
Great idea to go to Grenoble (France). There is there a nice ecosystem of companies + grad schools + researchers, one of the birth place of IT in France. If you are looking for the place to hire new engineers, or researchers to work on an advanced project, I would recommend it too.
- plam503711 1y ago
  That's entirely true (yeah I'm biased), frankly that's one of the best place in France for IT. And a lot cheaper/less stressful than Paris :p
ZeroTalent 1y ago
You wouldn't git clone a human
- plam503711 1y ago
  Someone got the ref, finally ;)
deleted 1y ago
[deleted]
xg15 1y ago
> even the capability to work with the UI/UX team... I had to chuckle.
constantcrying 1y ago
The median software developer in France (similar for e.g. Germany) makes around 50k. Software development as a path to personal wealth does not exist outside the US. The proposition by these companies for new hires is: "Put in a lot of work, longer hours and solve difficult problems, so that (after taxes) you can earn 2k more a year than someone with a far more relaxed job". The only people you are going to find are the very few ones intrinsically motivated to forego their social life in favor of your job offer and new graduates who don't know any better and will burn out and resent you and possibly the entire industry for the rest of their lives. I will not put in exceptional effort, unless you are offering me something exceptional in return. And no, the "opportunity" to work hard on some problem does not count. Basically all of Europe has this problem, companies are looking for people who put in exceptional work for average salaries. The problem isn't that these people don't exist or that you couldn't find them. The problem is that everyone who fits your role knows what kind of effort it involves and knows that their time will never be adequately rewarded.
e63f67dd-065b 1y ago
This kind of talent is ... not rare at all? And pretty easy to find too, just wander around the hallways at a major academic systems conference or hang around the kernel mailing lists, you'll meet most of the people working at the cutting edge of these things and get connected to those working on critical systems components. And yes, most of them work for FAANG or are funded by them. Seriously, book a plane ticket to ATC/OSDI, EuroSys, etc and talk to all the people there. The good ones are already hired by one of the big established players (FAANG, Red Hat, Intel, etc), which is why you need to offer competitive compensation to lure them away.

115 points by plam503711 1y ago

I think the issue started way before AI when it comes to not having enough fresh blood for this industry. Companies have decided they don't want to train juniors. They want seniors who already have the skills.

somanyphotons 1y ago
This just means the market is saturated. When there aren't enough people they'll accept less experience
- betterThanTexas 1y ago
  ...this doesn't make sense. Surely you need to factor in price point. Often times junior engineers deliver disproportionate value. Some ratio of juniors:seniors just seems rational, and those juniors grow into seniors. Maybe there's a good argument against training, but it could also just be irrational and stubborn in this case.
  - gbuk2013 1y ago
    In my experience it only works if you pull in juniors into strong teams and keep the proportion of juniors reasonably small. You also need to have a process in place for training them - it’s not enough to rely on ad-hoc mentoring from peers. If you get the ratio wrong, with too many juniors and weak technical leadership then you will end up in a very bad place in your code base. In terms of value, even if juniors are half the cost, it is much wiser to hire one senior instead of 2 juniors for the same money.
    - betterThanTexas 1y ago
      > it is much wiser to hire one senior instead of 2 juniors for the same money. Maybe in terms of pure productivity, but if you can match hiring to roadmaps you can give them more approachable/further from revenue work for which seniors would be an overkill. Etc. I'm just saying anyone with a simple explanation is only telling you part of the story. Besides, hiring is expensive. If you say live near a university, you have an edge in finding talent.
      - gbuk2013 1y ago
        > give them more approachable/further from revenue work for which seniors would be an overkill My experience suggests there is no such thing. There is work that seniors might not want to do, but if you hired well then they will be professional enough to do it, if it really needs to be done. And it takes some experience to determine which “non-revenue generating” work (e.g. tech debt) actually needs to be done, to advocate doing it to the stakeholders and to actually do it well. Juniors need a lot of supervision and that is not free. Which is not a reason not hire them in the first place, just that that it should be done mindfully.
        johnnyanmac 1y ago
        >which “non-revenue generating” work (e.g. tech debt) actually needs to be done This mentality is part of the problem. You can't fundamentally treat juniors as a profit center. Especially in more niche fields. They need to be trained and schools don't cover your specific pipeline. maybe it doesn't "need" to be done, but some refactoring work will pay off to turn your juniors today into seniors tomorrow. If you can only think in productivity charts, then you don't hire juniors.
- jandrewrogers 1y ago
  This doesn’t follow. Below some level of skill and experience they can contribute negative value to the project. Companies need a minimum level of experience just to make the role pay for itself. Does the high skill standard for surgeons mean the market for surgeons is saturated?
  - 9rx 1y ago
    If you are not needing to consider having someone who does not meet a certain skill standard to perform surgery on you, then, yes, it would seem that the surgeon market is saturated as the parent describes it.
    - phito 1y ago
      It depends on the time scale you are looking at it with. It looks saturated short-term, but it surely is not long term. The problem will only show up once your surgeons start retiring.
      - 9rx 1y ago
        Instantaneous time is the only time scale that makes sense with respect to the topic at hand. Sure, it is possible for the market to desaturate at some point as key people leave. But at that point, assuming suitable, per the earlier comment, replacements don't fill the void, the market will either come to accept lesser skilled people or it will come to accept fewer surgeries, returning to equilibrium. Not exactly magic.
n_ary 1y ago
> Companies have decided they don't want to train juniors. They want seniors who already have the skills. It got worse in last two years. Now Senior Engineers must have exact combo of the weird tech stack and tools with N years of experience exactly as the existing employees, else gtfo, you don't even get screening calls. You don't have something from nice to have list? Lol, why are you wasting our recruiter's time even? She needs to use GenAI to write her next rejection email. Also, your 15yoe does not matter, unless you are coming from direct competitor, in which case your 1.5yoe with internship is also excellent.
gbuk2013 1y ago
I’m a lead consultant on a gig where the CTO outright told us that he’s not really interested in growing their (frankly, under qualified) engineers, just wants them to get on with the job - it’s very short-sighted and sad. That said, in my previous job in a startup we hired very junior engineers and gave them plenty of opportunity and support for growth and several people did stunningly well. Pity the company didn’t do very well (IMHO due to focusing more or this sort of thing rather than making money). The truth is bound to be somewhere in the middle.
- htrp 1y ago
  >That said, in my previous job in a startup we hired very junior engineers and gave them plenty of opportunity and support for growth and several people did stunningly well. Pity the company didn’t do very well (IMHO due to focusing more or this sort of thing rather than making money). Mentoring is a key part of technical leadership, way easier to help the talent grow into your requirements (even if they are under-qualified, for now)
  - plam503711 1y ago
    I agree but there's also the extra difficulty to do mentoring remotely (we are a remote first company). I *really* like being remote first and provide the choice if you want to work on site or wherever you want. But it does come with some challenges.
    - johnnyanmac 1y ago
      This may be a hot take, but I don't think most juniors should be remote. You need some time face to face to understand the processes and you miss lot of passive knowledge when you're not navigating a workplace of people discussing matters. once you get a few years under the belt, sure. You're probably fine doing everything online. I can't think of many remote only studios hiring juniors anyways.
      - plam503711 1y ago
artyom 1y ago
I agree but it was always like that. Also there isn't any incentive anymore in growing juniors if they're going to immediately take the next better offer that comes around when they know enough, and you have such thing being offered on a daily basis (at least until a couple years ago). I would love to work on low-level, systems stuff (anything as close to the hardware as possible), that's even my education and area of expertise. BUT SaaS companies in reality: - pay better - have lower costs - are way easier - don't have that many (or any at all) geographical restrictions (e.g. importing hardware prototypes). People follows the money.
- johnnyanmac 1y ago
  >there isn't any incentive anymore in growing juniors if they're going to immediately take the next better offer that comes around when they know enough okay. Promote them with proper pay then. This HR debacle of putting more budget into hiring over retaining is entierly self-inflicted. >People follows the money. Well that love sounds more like a low priority whimsy in that case. I don't think tech workers of all people are ones who ever complain about low compensation. Unless you work in games, I suppose.
jandrewrogers 1y ago
I think it is important to acknowledge that it takes much longer to train juniors in systems software today than a couple decades ago because the nature of the problems have changed. It now takes several years to get someone to an acceptable level for a lot of systems software work. In many cases that’s longer than the entire development cycle — a junior would never really be productive on the project. And yes, this is creating a vicious feedback loop where we are no longer producing many new people with these skills despite a lot of demand. The minimum level of sophistication required to be effective in systems software has increased dramatically since the 1990s, when I first started doing systems software. The kinds of systems software we were putting in production in back then would be considered a trivial toy today. This shift has placed an enormous amount of upward pressure on the minimum level of experience and skill that would allow someone to become productive within a useful amount of time. It is no longer feasible in many cases to have companies effectively subsidize the development of highly skilled systems software people. The training time has become so long that companies will never recoup the investment. It is easy to see how the incentives have created the current situation and it is not clear how to address the shortage. Even before the current situation, it was widely noted that most systems software developers were self-taught over many years rather than trained up in a structured environment.
- johnnyanmac 1y ago
  >And yes, this is creating a vicious feedback loop where we are no longer producing many new people with these skills despite a lot of demand. if the company would rather burn out and age out all their talent than invest in the future, I don't know what to say. For niche domains, you can't treat talent a a pure business expense. That's how you lose out all that talent to China and suddenly you can't compete at all. If anything, training should be treated like insurance. >It is no longer feasible in many cases to have companies effectively subsidize the development of highly skilled systems software people. "subsidize" implies that somehow engineers are saving money by choosing to take these more niche paths. It doesn't make sense. You have a product and you need talent to develop and maintain it to compete. There's no "subsidies" in a business relationship like this. your choices are few and simple: train in-house, lobby schools to train for you, or publicize your tech and hope people train themselves for you again. It seems like these days companies aren't looking out long term though and just want to jump the ship that has plenty of time to steer past the iceberg everyone sees ahead.

tra3 1y ago

Maybe I'm being too cynical, but I can't tell whether this is a promotional piece for something.. That said, I do agree with the premise of the article that it's hard to learn "the stack", especially with the advent of generative AI. "Back in the day", when google spat out a link to something resembling your problem, you still had to deconstruct the answer given to apply it to your particular case. That required some sort of understanding, or a mental model of the domain. With ChatGPT I find myself copying/pasting error messages between two browser windows, or a terminal, not really understanding what is going on. No learning is happening. Does not bode well for new folks coming in.

dingnuts 1y ago
doesn't bode well for you either
- tra3 1y ago
  "A problem well stated is a problem half solved", is all I got for now.
plam503711 1y ago
Hi! It's not a promotional piece of something, it's my personal experience as a CEO and co-founder of a company using Xen as the core of our stack. I like to share my views in a transparent fashion on how it's hard to do very technical stuff, not just for technical reasons, but due to the lack of people trained for.
maximumgeek 1y ago
If you find that to be the case when you use ChatGPT, maybe you should quit using ChatGPT. Sometimes, probably most of the time, it is better to work through it and understand an issue than to blindly copy pasta.
pier25 1y ago
> No learning is happening. Does not bode well for new folks coming in. Recently got into a completely new language/framework and used Copilot to understand what was going on. I still made all the big decisions and wrote most of the code myself. You can definitely use AI to avoid learning though (up to a point).

throw093209 1y ago

> System teachers: now rarer than RGB-free laptops LLM are quite good at explaining systems and frameworks. I would never got into kernel programming without Deepseek guidance. As for universities: too expensive, too much paperwork, too slow, too elitist.

plam503711 1y ago
It's true but until some extent. When you are talking about a hypervisor (like Xen), and many many subtle things depending on your CPU brand/model, it's really really *hard*, even with an LLM (and even more with an LLM hallucinating some CPU features or forgetting basic things like Meltdown and Spectre). However I agree: to learn a topic, LLMs are providing a great speedup. As a CEO/co-founder, I have no issue to hire people without a degree if they are good at what they do. However, our biggest chances are to scout directly in universities to find motivated students (motivation >>> everything else)
- throw83394848 1y ago
  As a CEO, are you going to hire that motivated student? Or is it some sort of unpaid internship?
  - plam503711 1y ago
    We do not have any unpaid internship. So yes, I will hire them, until we reach a limit of mentors for them, because we cannot leave them alone.

citizenpaul 1y ago

Somebody tell that to the MBA's.

thenewwazoo 1y ago

As an aside, TFA says: We’re talking about skills that span kernel-level programming, hardware quirks, low-level debugging, distributed systems, security, orchestration logic, even the capability to work with the UI/UX team... and the ability to explain all that without scaring interns. You can’t just hire for that. You have to grow it. Nurture it. Beg for it. Or in some cases, resurrect it. If you are that person, what is the best way to market yourself? I am the person described. I've got experience from poking registers in firmware, to wireline transport protocol implementation, to infosec, to writing microservice framework middleware, to pipeline orchestration at the OS level, and on and on. In the last week I've debugged Linux UDS issues and TLS cipher suite problems, and wrote code to provision WiFi-connected devices over BLE. But it's incredibly hard to demonstrate that in an interview, if I can even find a role that warrants it. You're not going to find me on a university campus or in a research lab because I'm at a FAANG trying to pay my mortgage.

frainfreeze 1y ago
I found it is best to keep quiet, not even have a (directly attached to your name) blog with such varried content, and instead just send appropriate version of your CV when needed. When it comes to interviews I found it helps that I'm in my office/lab space, able to pick up the cam and show the person on call contents of my shelves and present few details about them on the office whiteboard. But even then they of course doubt you. Tldr raise your price and they'll belive easier.
- ziddoap 1y ago
  The question was "what is the best way to market yourself?" and your recommendation is to keep quiet and do less marketing. If that has worked for you, that's amazing! But it seems really counter-intuitive to me.
  - zeta0134 1y ago
    I think the counterintuitive thing is that the marketing communicates a lack of confidence. It's... sortof odd to think about it that way, but to communicate actual confidence in your field, you mostly have to be willing and able to have a conversation about whatever topic your interviewer fancies. Being able to do that comfortably speaks volumes that your resume and project portfolio struggle to replicate.

neilv 1y ago

> What we’re doing about it (before it’s too late) I don't see "pay like Google" listed.

plam503711 1y ago
Because we cannot afford that, we aren't Google, and still a relatively small company vs the task of building a full-stack virtualization solution. Luckily, we have other strong points helping a lot (remote first, no micro-management, a great culture promoting human values and so on etc.)

koala_man 1y ago

The article starts off saying that if you want people with real full stack experience, from kernel to UX, you need to grow it. It goes on to say that it's hard to find and develop expertise for low level software like hypervisors. What's the connection between the topics? It feels like two different rants. If it's difficult to find kernel developers then wouldn't it help to not require them to also know web UX?

ajsnigrutin 1y ago
> If it's difficult to find kernel developers then wouldn't it help to not require them to also know web UX? That means hiring two people, and in $current_year, companies expect one person to know everything. Sysadmin, backend programmer, frontend programmer, designer and a DBA used to be different people not that long ago, now they expect one person to do all that... + it seems they want kernel development experience now.
- delusional 1y ago
  Before they were multiple people they were one person. A single person can in fact write a program for a computer.
  - ajsnigrutin 1y ago
    Sure, some C code, some html, a table here, a colspan there, and you can have a website made by a single person... if we want a website to look like it was made on an 1980s computer by a single person.
    - wizzwizz4 1y ago
      Or, you can have a decent website made by a single person. It's not that hard to learn basic HTML5, enough ARIA semantics to know not to use ARIA, a programming language with decent synchronisation primitives that supports CGI, an SQL dialect and the principles of relational database design, enough JavaScript to use MDN, enough CSS to use MDN, the basics of server maintenance, TLS certificate provision, and DNS. If you want to do your own networking, or run email, that's a whole 'nother specialism; but just running a website is easy enough for one person to do.

smileysteve 1y ago

The title holds for much more specific and thinner stacks, along with support, QA, and sales - especially even simple web apps. Onboarding is important Training is important Retaining is important Maybe your system is 100% documented, conventional (looking at you Rails, Angular), debt free, tested, instrumented - but more than likely it's not. But if you get down to staff who can't teach a a system, including product teams that don't respect feature overload, value, or internal feature training. If you prioritize new features over team development and training (aka a team that doesn't know the system), you're likely to get muddy with existing features both technically and use facing.

plam503711 1y ago
I agree. If it's already hard for simpler stacks, you can imagine how hard it is for more critical or complex ones. And it's even worse if you inherited some of it (ie collecting technical debt that's not yours, which is the case here as some part of the stack are the result of a fork). Sometimes it's even a catch-22 situation, where the technical/generic knowledge is already hard to find, but you absolutely need it to train more junior people. Luckily we found such experimented people, but then you also need to use their expertise to actually fix stuff and not just mentor juniors. A very very delicate balance to find, especially in a timed market.
mystified5016 1y ago
My boss is about to learn this lesson the hard and painful way. Retention has fallen to the point where I am the only person who groks the full stack. I'm making an attempt to document what I can, but they're going to be fucked when I leave. I would feel bad about it, but CEO has tried to fuck me over many times-- for no apparent reason.

lesser-shadow 1y ago

>Partnering with Universities and Research Labs Like where? In the US? Western Europe? You're cleary not searching hard enough because I've never heard about your company once throughout my time at university.

johnnyanmac 1y ago
They shoutout a city in France, so I assume western EU.

deleted 1y ago

[deleted]

gitroom 1y ago

been there with bosses not caring about team learning at all, kinda messes everything up long run tbh

lammalamma25 1y ago

Curious if there is a way to dip your toe into this kind of thing without dedicating years of learning. Maybe a home lab? I've always been interested in this because it seems hard/interesting, and enjoyed learning about it in school. Wondering how much time you'd have to invest to see if you like it. I assume there is a healthy amount of "this is hard/not fun" along the way before you find out.

cadamsdotcom 1y ago

The value of knowledge is plummeting. It can, will, and should be subsumed by LLMs. Consider London’s black cabs. Drivers pass a gruelling exam to prove they know “everything” about one part of London. But, Google Maps puts that knowledge and more on every rideshare driver’s dashboard, for free, no advance study needed. Map knowledge doesn’t make you able to drive. Driving happens in real time - steering, obeying road rules, avoiding accidents. In engineering, the real time part is when you use mental models you’ve painstakingly developed - how the hardware works so you can debug it, what syntax is valid in your programming languages, what APIs exist in your stack. But, those depend on knowledge - so mental models can be self-taught with AI as tutor in your own time. The LLMs supply the knowledge, you integrate it via study - or a side project. An organization cannot clone itself a team, it’s true. They need engineers now, not in 6-12 months. But - a motivated individual can use AI to make themselves employable as a full stack engineer faster now than at any point in history. How cool is that!

jandrewrogers 1y ago
One of the big issues with LLMs for e.g. systems software is that there are broad domains where training data effectively does not exist. Consequently, you can’t learn much from them of value. It is the blind leading the blind. The lack of training material for LLMs is of course a lack of training material for people too. Some areas of software have a long history of relying almost entirely on an oral tradition to pass down knowledge. This has some advantages but it doesn’t scale and it makes it basically “dark knowledge” for LLMs or people without access to those that know it. If you want to get into an area like this, you often need to find a way to spend a lot of time with people that already know it.
- johnnyanmac 1y ago
  >This has some advantages but it doesn’t scale and it makes it basically “dark knowledge” for LLMs or people without access to those that know it. We call this "tribal knowledge" in games. I despise it. It's done for a few reasons: - NDAs make public knowledge a landmine. And every game studio makes you sign NDAs. Even at the interview stage. - Churn. No one gets time to really develop expertise as they work on a project for 2-3 years and then layoffs come. Only a relative few become experts, and probably not because of the studio itself - lack of incentives. Games aren't very connected with acedemia to begin with, despite relying so much on cutting edge tech. So the best resources for sharing such techniques is shafted. This is slowly getting better as more tech conferences talk about games tech, but it's a pretty slow tricke unless you come from one of the largest studios and specifically come in for R&D. >If you want to get into an area like this, you often need to find a way to spend a lot of time with people that already know it. All too true. Open source development is one bastion for this, but that's overall why I keep trying to stay in this domain. You literally can't get the knowledge elsewhere. And it's knowledge that directly leads to better looking, more optimal, and less buggy games overall.

ygouzerh 1y ago

Great idea to go to Grenoble (France). There is there a nice ecosystem of companies + grad schools + researchers, one of the birth place of IT in France. If you are looking for the place to hire new engineers, or researchers to work on an advanced project, I would recommend it too.

plam503711 1y ago
That's entirely true (yeah I'm biased), frankly that's one of the best place in France for IT. And a lot cheaper/less stressful than Paris :p

ZeroTalent 1y ago

You wouldn't git clone a human

plam503711 1y ago
Someone got the ref, finally ;)

deleted 1y ago

[deleted]

xg15 1y ago

> even the capability to work with the UI/UX team... I had to chuckle.

constantcrying 1y ago

The median software developer in France (similar for e.g. Germany) makes around 50k. Software development as a path to personal wealth does not exist outside the US. The proposition by these companies for new hires is: "Put in a lot of work, longer hours and solve difficult problems, so that (after taxes) you can earn 2k more a year than someone with a far more relaxed job". The only people you are going to find are the very few ones intrinsically motivated to forego their social life in favor of your job offer and new graduates who don't know any better and will burn out and resent you and possibly the entire industry for the rest of their lives. I will not put in exceptional effort, unless you are offering me something exceptional in return. And no, the "opportunity" to work hard on some problem does not count. Basically all of Europe has this problem, companies are looking for people who put in exceptional work for average salaries. The problem isn't that these people don't exist or that you couldn't find them. The problem is that everyone who fits your role knows what kind of effort it involves and knows that their time will never be adequately rewarded.

e63f67dd-065b 1y ago

This kind of talent is ... not rare at all? And pretty easy to find too, just wander around the hallways at a major academic systems conference or hang around the kernel mailing lists, you'll meet most of the people working at the cutting edge of these things and get connected to those working on critical systems components. And yes, most of them work for FAANG or are funded by them. Seriously, book a plane ticket to ATC/OSDI, EuroSys, etc and talk to all the people there. The good ones are already hired by one of the big established players (FAANG, Red Hat, Intel, etc), which is why you need to offer competitive compensation to lure them away.