37 ms·

Oh lordy, the "two crews" bifurcation fully written down. What a fantastic way to ship until it becomes far too expensive to ship anything good. Look, when we

by m104 2y ago

Oh lordy, the "two crews" bifurcation fully written down. What a fantastic way to ship until it becomes far too expensive to ship anything good. Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything. Worse, your feature crews aren't learning anything beyond how to get those lines out the door, which will somehow get slower and more expensive as time goes on. Why? Because you removed the one fitness function on good software development which is to fully re-incorporate the negative feedback back into the source of development. A real CTO leadership handbook would say clearly "it's your responsibility to help your developers improve, especially while shipping, and they're not always going to be happy about it."

borski 2y ago
One thing that we did to account for this was to shift the teams every or every few sprints. It allowed folks to get more experience, still get feedback, since if they built a buggy feature they'd have to fix it, etc. People seemed much happier with that, because they also didn't get tired of 'always fixing bugs' or never getting the feedback, which you insightfully mentioned.
bboreham 2y ago
I call them “shiny team and shitty team”.
DanielHB 2y ago
Related topic, but every company I worked at that had a platform team (as in a third-crew support team that manages tools/practices/common-code for a discipline) ends up being infested with over-engineering. They tend to attract that kinda of people who have disdain about delivering features and fixing bugs and like to over-abstract problems. Instead of fixing bugs they try to create increasingly complex abstractions to prevent the bugs from happening in the first place, with obvious results.
- actionfromafar 2y ago
  I wonder though if there aren't more forces at play. For instance, the business problems some systems try to solve really are so large and complex, you might need some kind of overseeing function in your company. Also I have a hunch a team dedicated to providing helper "libraries" more than than "frameworks" could provide a lot of value without so much downside. If you can call a library function without it imposing a whole framework on the rest of your codebase, it's more self-contained and can't spill its abstractions all over the place.
  - DanielHB 2y ago
    If your org starts a platform team it is really important to have this concept drilled on early. Buffet, not framework. I clearly remember having some discussions with platform people in my last job and asking them "why should I use your solution instead of getting an open source one that is likely better tested and used by more people" and the answer was usually "we can help if you run into any problems". Well, the "help" is to be planned and prioritized in the next sprint and probably will only come next quarter. So now the devs in my team need to make PRs to the platform people code and beg for reviews, how is that better than using the open source?
- neumann 2y ago
  argh! PTSD - This was exactly what happened at my last start-up. Two of the engineering team and one from the R&D team started a platform team and it became a pre-PMF product with the slickest pipelines, DevOps, Cloud-cost optimization ready to scale to infinity. But with no customers, a broken front-end, and a under-funded R&D team as all the effort was put into the essential SaaS Platform. Truly put the company back 1 year while burning two.
  - DanielHB 2y ago
    That is actually usually not that bad (if there is, you know, revenue). What is really bad is when those teams start to roll out a lot of custom code that other teams need to use. If they are just configuring standard tools for everyone else it is usually fine (as long as they are not going to crazy with it).
pyrale 2y ago
> It allows your feature team to remain 100 percent focused on the future, undistracted by customer support work. AKA "it allows your feature team to be completely oblivious to the horrors they unleash, and keep at it until the ship is solidly planted in the iceberg" Not talking about the conflicts it creates for merging between sales-supported feature teams and customer rep-supported maintenance teams. Given that the "customer crew" is described as something you grow out of, there's no question who wins arbitrages. > It provides another career path for individual engineers, especially junior engineers, to learn and level up on your team. "Senior staff doesn't want to fix shit so we have juniors do it"
- n4r9 2y ago
  Further down the article: > The Microsoft blog post referenced above recommends swapping some team members between the two crews every week. This would hopefully mitigate the worst of the effect you describe, since everyone eventually gets exposed to the consequences of poor feature development.
  - pc86 2y ago
    I don't know about you but it's rare that I've neatly wrapped up my tasks at the end of any given week. Single-day tasks are rare, there is always carry-over work including over the weekends. The only thing worse than a feature that got rushed out the door Friday afternoon because you had a completely different role come Monday is one that was 80% done then passed off to someone else because you had a completely different role come Monday.
    - BossingAround 2y ago
      In my company, we rotate every 5 months. So every 6th month, I get put into the customer-facing team for 1 month. Every other month, a different team member is on the customer-facing team. This is still annoying, but gives you enough time to work on features, and enough time to try and crack some customer cases (though I could even see being in the customer-facing team for more than 1 month, as sometimes, this is not enough to debug the issue and provide a fix). I've got to admit, as much as I dislike being on the customer team, it's certainly less annoying than working on features, and have constant customer issues interruptions though.
madeofpalk 2y ago
Developers must run and maintain the software they build. It's as simple as that.
- x0x0 2y ago
  And take the calls. Don't like being paged at 3am? Write robust software and test.
- dowager_dan99 2y ago
  The best development teams WANT this. They will readily take on the responsibility to get the autonomy. The problem is many companies give the former without the latter...
JohnCClarke 2y ago
Well, what's the time horizon? A PE backed outfit, or a CTO looking to move on within a year or so, would be well advised to follow this guidance. Lots of success now, and the problems deferred to later.
bravetraveler 2y ago
This may put me and my peers out of work (in a good way). SRE is a consequence of this function being lost, IMO. Pattern: developers don't like it? Give it to Ops/SRE. Take away the escape, we will all be better for it.
plomme 2y ago
The book mentions having a rota: > Engineers rotate between the crews on a regular basis. The Microsoft blog post referenced above recommends swapping some team members between the two crews every week. In my experience this works well. With my current and previous client each team had a "hero of the week", whose responsibility was second line support and monitoring. If nothing came up the hero would work on their tasks as usual. If something does come up the heroes of the week would be tasked with solving it or pulling in someone who knows how to solve it. This leads to engineers both having to accept accountability for writing shoddy code, but it also exposes engineers to the wider codebase when pulling on threads. It also solves the issue where no-one or the same person always takes responsibility for handling bugs.
- marcinzm 2y ago
  Isn't this just called on-call? That's very different from a separate team.
  - plomme 2y ago
    Maybe - though I associate being "on-call" with being expected to respond outside of normal business hours which was not the case in the teams I worked in.
- dowager_dan99 2y ago
  This just sounds like having a point developer. The challenge is too many companies expect this without giving up a feature-dev headcount. Any work the get done aside from point is a bonus and unplanned.
hnthrow90348765 2y ago
>Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything. This is the PM's job - one or a few people who are deciding the vision of how all of the features fit together based on feedback by working with customers. Customers (esp. non-technical ones) will definitely not have a coherent product vision and only want immediate fixes regardless of what else may be planned. Customers may also not communicate to one another and their feedback can conflict. If you put this burden on developer shoulders, they now have to manage all of that communication in addition to requiring technical skills to know the code base and maintain it well, on top of every developer needing to have the same coherent vision to make thoughtful decisions. That's now two to three jobs in one depending if your developers also manage infrastructure like many roles are requiring these days.
- marcinzm 2y ago
  What you're describing is exactly the opposite of every actually successful team I've seen, and describes every mediocre team I've seen. Silos are death and not just in a code base. Good developers understand the product. Mediocre ones churn out tickets mindlessly.
  - hnthrow90348765 2y ago
    I'm okay with that knowing those developers are doing two jobs for the pay of one. And most products turn into that once the original developers leave. It's not like you can't learn the product through the PM either.
    - pc86 2y ago
      I think you're conflating "doing two jobs" with "not being allowed to just type JavaScript into a computer all day in isolation and being expected to actually communicate and think about things other than data structures and algorithms." If you're a true senior software engineer as most of us claim to be coding is a small part of your job, not your entire job. You should be learning the product through the PM for sure, and I don't think a senior engineer should be doing first-level support, but especially in small companies talking to customers is good and should be expected from basically everyone who is working on the product.
esafak 2y ago
It is not a problem if you measure and reward the infra team for their ability to enable the feature team, such as change lead time and deployment frequency, as well as the the stability metrics that the infra team might want to pursue.

by m104 2y ago

borski 2y ago

One thing that we did to account for this was to shift the teams every or every few sprints. It allowed folks to get more experience, still get feedback, since if they built a buggy feature they'd have to fix it, etc. People seemed much happier with that, because they also didn't get tired of 'always fixing bugs' or never getting the feedback, which you insightfully mentioned.

bboreham 2y ago

I call them “shiny team and shitty team”.

DanielHB 2y ago

Related topic, but every company I worked at that had a platform team (as in a third-crew support team that manages tools/practices/common-code for a discipline) ends up being infested with over-engineering. They tend to attract that kinda of people who have disdain about delivering features and fixing bugs and like to over-abstract problems. Instead of fixing bugs they try to create increasingly complex abstractions to prevent the bugs from happening in the first place, with obvious results.

actionfromafar 2y ago
I wonder though if there aren't more forces at play. For instance, the business problems some systems try to solve really are so large and complex, you might need some kind of overseeing function in your company. Also I have a hunch a team dedicated to providing helper "libraries" more than than "frameworks" could provide a lot of value without so much downside. If you can call a library function without it imposing a whole framework on the rest of your codebase, it's more self-contained and can't spill its abstractions all over the place.
- DanielHB 2y ago
  If your org starts a platform team it is really important to have this concept drilled on early. Buffet, not framework. I clearly remember having some discussions with platform people in my last job and asking them "why should I use your solution instead of getting an open source one that is likely better tested and used by more people" and the answer was usually "we can help if you run into any problems". Well, the "help" is to be planned and prioritized in the next sprint and probably will only come next quarter. So now the devs in my team need to make PRs to the platform people code and beg for reviews, how is that better than using the open source?
neumann 2y ago
argh! PTSD - This was exactly what happened at my last start-up. Two of the engineering team and one from the R&D team started a platform team and it became a pre-PMF product with the slickest pipelines, DevOps, Cloud-cost optimization ready to scale to infinity. But with no customers, a broken front-end, and a under-funded R&D team as all the effort was put into the essential SaaS Platform. Truly put the company back 1 year while burning two.
- DanielHB 2y ago
  That is actually usually not that bad (if there is, you know, revenue). What is really bad is when those teams start to roll out a lot of custom code that other teams need to use. If they are just configuring standard tools for everyone else it is usually fine (as long as they are not going to crazy with it).

pyrale 2y ago

> It allows your feature team to remain 100 percent focused on the future, undistracted by customer support work. AKA "it allows your feature team to be completely oblivious to the horrors they unleash, and keep at it until the ship is solidly planted in the iceberg" Not talking about the conflicts it creates for merging between sales-supported feature teams and customer rep-supported maintenance teams. Given that the "customer crew" is described as something you grow out of, there's no question who wins arbitrages. > It provides another career path for individual engineers, especially junior engineers, to learn and level up on your team. "Senior staff doesn't want to fix shit so we have juniors do it"

n4r9 2y ago
Further down the article: > The Microsoft blog post referenced above recommends swapping some team members between the two crews every week. This would hopefully mitigate the worst of the effect you describe, since everyone eventually gets exposed to the consequences of poor feature development.
- pc86 2y ago
  I don't know about you but it's rare that I've neatly wrapped up my tasks at the end of any given week. Single-day tasks are rare, there is always carry-over work including over the weekends. The only thing worse than a feature that got rushed out the door Friday afternoon because you had a completely different role come Monday is one that was 80% done then passed off to someone else because you had a completely different role come Monday.
  - BossingAround 2y ago
    In my company, we rotate every 5 months. So every 6th month, I get put into the customer-facing team for 1 month. Every other month, a different team member is on the customer-facing team. This is still annoying, but gives you enough time to work on features, and enough time to try and crack some customer cases (though I could even see being in the customer-facing team for more than 1 month, as sometimes, this is not enough to debug the issue and provide a fix). I've got to admit, as much as I dislike being on the customer team, it's certainly less annoying than working on features, and have constant customer issues interruptions though.

madeofpalk 2y ago

Developers must run and maintain the software they build. It's as simple as that.

x0x0 2y ago
And take the calls. Don't like being paged at 3am? Write robust software and test.
dowager_dan99 2y ago
The best development teams WANT this. They will readily take on the responsibility to get the autonomy. The problem is many companies give the former without the latter...

JohnCClarke 2y ago

Well, what's the time horizon? A PE backed outfit, or a CTO looking to move on within a year or so, would be well advised to follow this guidance. Lots of success now, and the problems deferred to later.

bravetraveler 2y ago

This may put me and my peers out of work (in a good way). SRE is a consequence of this function being lost, IMO. Pattern: developers don't like it? Give it to Ops/SRE. Take away the escape, we will all be better for it.

plomme 2y ago

The book mentions having a rota: > Engineers rotate between the crews on a regular basis. The Microsoft blog post referenced above recommends swapping some team members between the two crews every week. In my experience this works well. With my current and previous client each team had a "hero of the week", whose responsibility was second line support and monitoring. If nothing came up the hero would work on their tasks as usual. If something does come up the heroes of the week would be tasked with solving it or pulling in someone who knows how to solve it. This leads to engineers both having to accept accountability for writing shoddy code, but it also exposes engineers to the wider codebase when pulling on threads. It also solves the issue where no-one or the same person always takes responsibility for handling bugs.

marcinzm 2y ago
Isn't this just called on-call? That's very different from a separate team.
- plomme 2y ago
  Maybe - though I associate being "on-call" with being expected to respond outside of normal business hours which was not the case in the teams I worked in.
dowager_dan99 2y ago
This just sounds like having a point developer. The challenge is too many companies expect this without giving up a feature-dev headcount. Any work the get done aside from point is a bonus and unplanned.

hnthrow90348765 2y ago

>Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything. This is the PM's job - one or a few people who are deciding the vision of how all of the features fit together based on feedback by working with customers. Customers (esp. non-technical ones) will definitely not have a coherent product vision and only want immediate fixes regardless of what else may be planned. Customers may also not communicate to one another and their feedback can conflict. If you put this burden on developer shoulders, they now have to manage all of that communication in addition to requiring technical skills to know the code base and maintain it well, on top of every developer needing to have the same coherent vision to make thoughtful decisions. That's now two to three jobs in one depending if your developers also manage infrastructure like many roles are requiring these days.

marcinzm 2y ago
What you're describing is exactly the opposite of every actually successful team I've seen, and describes every mediocre team I've seen. Silos are death and not just in a code base. Good developers understand the product. Mediocre ones churn out tickets mindlessly.
- hnthrow90348765 2y ago
  I'm okay with that knowing those developers are doing two jobs for the pay of one. And most products turn into that once the original developers leave. It's not like you can't learn the product through the PM either.
  - pc86 2y ago
    I think you're conflating "doing two jobs" with "not being allowed to just type JavaScript into a computer all day in isolation and being expected to actually communicate and think about things other than data structures and algorithms." If you're a true senior software engineer as most of us claim to be coding is a small part of your job, not your entire job. You should be learning the product through the PM for sure, and I don't think a senior engineer should be doing first-level support, but especially in small companies talking to customers is good and should be expected from basically everyone who is working on the product.

esafak 2y ago

It is not a problem if you measure and reward the infra team for their ability to enable the feature team, such as change lead time and deployment frequency, as well as the the stability metrics that the infra team might want to pursue.