8 ms·

A more common scenario I see is that people start with a monolith that ends up inheriting all the conceptual debt that accumulates as a project evolves. All thi

by ser0 6y ago

A more common scenario I see is that people start with a monolith that ends up inheriting all the conceptual debt that accumulates as a project evolves. All this debt builds up a great desire for change in the maintaining team. A champion will rise with a clean architecture and design in microservice form that addresses all high visibility pain points, attributing forecasted benefits to the perceived strengths of microservices. The team buys into the pitch and looks forwards to a happily-ever-after ending. The reality though is that the team now has multiple problems, which include: - Addressing conceptual debt that hasn't gone away. - Discovering and migrating what the legacy system got right, which is often not documented and not obvious. - Dealing with the overheads of microservices that were not advertised and not prominent at a proof-of-concept scale. - Ensuring business continuity while this piece of work goes on. I would propose alternative is to fix your monolith first. If the team can't rewrite their ball of mud as a new monolith, then what are the chances of successfully rewriting and changing architecture? Once there is a good, well functioning monolith, shift a subset of responsibility that can be delegated to a dedicated team - the key point is to respect Conway's law - and either create a microservice from it or build a new independent monolith service, which aligns more to service oriented architecture than microservices.

zimpenfish 6y ago
Another issue I've seen is that people push all the problems onto the monolith even if they're external - one place had a Perl monolith (which was bad, sure) but their main issue was an overloaded database which could have been addressed with moving some queries out of the (awful homegrown) ORM and using e.g. signed session cookies instead of every request causing a session table hit.
Chyzwar 6y ago
There is one approach Fowler suggested is SacrificialArchitecture. You build your monolith quickly, get to market fit and once you understand service boundaries you move to microservices. Personally I would like to try Umbrella Projects[1]. You can design it as microservices but deploy and build as monolith. Overhead is lower, and it is easier to figure out right services when in one codebase. It can be easy implemented in other lang/frameworks as well. [1] https://elixirschool.com/en/lessons/advanced/umbrella-projects/ https://elixirschool.com/en/lessons/advanced/umbrella-projec...
- andy_ppp 6y ago
  This! Service boundaries are vital and almost intractable to design up front as you won’t be sure of your systems’ use cases. I’ve worked on numerous micro services systems and all of them were designed in such a way that data loss and unknown error states were mandatory. Micro services seem simpler, but are actually harder than my methodology “as few services as necessary” + bounded contexts within them. Using network partitions to stop spaghetti code just leads to spaghetti network partitions. The discipline to keep this simple and self contained is the important part.
  - radicalbyte 6y ago
    This is why Domain Driven Design should be mandatory reading for all anyone making decisions on Software Architecture.
    - datavirtue 6y ago
      Agreed. But do we mandate that they understand it and somehow enforce their implementation competency? As pervasive and helpful and as easy to understand what Clean Code is, it is still very rare to find someone who understands it and applies it in the field.
    - hackerfromthefu 6y ago
      Unfortunately in actual practise I've encountered significant cargo culting about DDD. Attracts a lot of people with mid level experience who suddenly want to dictate architecture based on theoretical ideals rather than practicalities. There's no substitute for experience, and specifics of adapting architecture to the context of the problem you're trying to solve. In one case I wanted to use a technology that actually matches DDD very significantly - but the cargo cultish closedmindedness of the practitioners meant they couldn't even understand how an old idea/tech they hadn't liked or approved was, was actually an implementation of DDD. The problem there is not DDD, the problem is the people who get closed minded and stuck to their one true way (often without really broad experience to make that judgement call effectively). I've learned that pattern of language of absolutes such as 'should be, mandatory, all' do XXX is often a sign of that kind of cargo cultish thinking.
jtcto 6y ago
I really like how you've worded this and agree. Some of this is driven by well documented cognitive biases, such as the availabily heuristic which you've identified, but there are so many. This is a good video on it https://birgitta.info/redefining-confidence/ https://birgitta.info/redefining-confidence/
tutfbhuf 6y ago
> If the team can't rewrite their ball of mud as a new monolith, then what are the chances of successfully rewriting and changing architecture? Well sometimes there are very complex subsystems in the monolith and it's easier to create a completely new microservices out of that instead of trying to rewrite the existing code in the monolith. We had done so successfully by creating a new payment microservices with stripe integration and then just route every payment that way. Doing the same in a huge pile of perl mess has been assessed as (nearly) impossible by the core developer team without any doubts. But I have to admit that the full monolith code base is in a maintenance mode beyond repair, only bug & security fixes are accepted at this point in time. Feature development is not longer a viable option for this codebase.
pjmlp 6y ago
When the same practices used to sell microservices get applied to modules, packages and libraries, there is no need to put a network in the middle to do the linker's job. But too many are eager to jump into distributed computing without understanding what they are bring into their development workflow and debugging scenarios.
- tutfbhuf 6y ago
  When I started programming, linking was the only way of packaging software in mainstream computing, if it was so superior we wouldn't have moved away from it.
  - pjmlp 6y ago
    Here is the thing, whatever you write doesn't work without linking, so no we never moved away from it.
    - diroussel 6y ago
      But is linking the right way to couple business logic from two different organisational units? That is the question being discussed.
      - tutfbhuf 6y ago
        > But is linking the right way to couple business logic from two different organisational units I think it's not, but I would like to hear other opinions.
        diroussel 6y ago
        I'm not sure why you are being down voted. In context of calling business logic we have moved away from linking. I guess the context is only implelied in your parent post. The rise of the network API in the last 20 years has proven it's own benefits. Whether you are calling a monolith of a microservice, it's easier to upgrade the logic without recompiling and re-linking all dependencies.
iainmerrick 6y ago
I think the key point of Fowler’s article, which is obscured a little by all the monolith discussions, is that try to start with microservices doesn’t work. He’s claiming this from an empirical point of view -- if you look at successful, working systems, they may use microservices now but they most often started as monoliths. People are talking about Conway’s law, but the more important one here is Gall’s law: “A complex system that works is invariably found to have evolved from a simple system that worked.”
- dboreham 6y ago
  Conway/Gall duality.
darkstar_16 6y ago
This is the correct way to do things but, unfortunately, like I wrote in my other comment, this knowledge only comes after one has gone through a few of these "transformational" changes. I've seen this happen a few times and monolith vs micro services is such a done deal that it's very difficult to argue against so, even with experience, you will always fail if you try and go against the grain :)
D_Guidi 6y ago
> I would propose alternative is to fix your monolith first I'd like the idea of vertical slices for this: once you've done this, then refactor to microservices will be much easier.
distalx 6y ago
We are just moving complexity from one place to another. And sometimes few things get lost in this transformation.
jimbokun 6y ago
> I would propose alternative is to fix your monolith first. If the team can't rewrite their ball of mud as a new monolith, then what are the chances of successfully rewriting and changing architecture? Often, the problems with the monolith are * different parts of the monolith's functionality need to scale independently, * requirements for different parts of the monolith change with different velocity, * the monolith team is too large and it's becoming difficult to build, test, and deploy everyone's potentially conflicting changes at a regular cadence, with bugs in one part of the code blocking deploying changes to other parts of the code. If you don't have one of these problems, you probably don't need to break off microservices, and just fixing the monolith probably makes sense.
- stepbeek 6y ago
  > the monolith team is too large and it's becoming difficult to build, test, and deploy everyone's potentially conflicting changes at a regular cadence, with bugs in one part of the code blocking deploying changes to other parts of the code. This is the only argument I'm ever really sold on for an SOA. I wonder if service:engineer is a ratio that could serve as a heuristic for a technical organization's health? I know that there are obvious counter-examples (shopify looks like they're doing just fine), but in other orgs having that ratio be too high or too low could be a warning sign that change is needed.
- Mavvie 6y ago
  > different parts of the monolith's functionality need to scale independently, I hear this a lot. Can you give me an example? Can't I just add more pods and the "parts" that need more will now have more? Agreed on the other reasons (although ideally the entire monolith should be CI/CD so release cadence is irrelevant, but life isn't perfect)
  - jimbokun 6y ago
    So the alternative is to have a monolith deployed with different "identities" based on how it's configured. So you configure and scale out different groups of the monolith code independently, to satisfy different roles and scale up or down as needed. However, sometimes the resources needed to load at start up can be drastically different, leading to different memory requirements. Or different libraries that could conflict and can also impact disk and memory requirements. For a large monolith, this can be significant. So at what point do you go from different "configuration", to where enough has changed it's a truly different service? The dividing line between code and configuration can be very fluid. > the entire monolith should be CI/CD so release cadence is irrelevant But if one module has a bug and is failing testing, or features partially completed, it blocks releasing all the other code in the monolith that is working.
tootie 6y ago
I'm living this right now. A big, fat python 2 monolith maintained for 10 years. Everyone hates it, but it actually works perfectly fine.
spaetzleesser 6y ago
That’s what I am always saying. If you can’t manage libraries you will fail at microservices too, just harder. I am working on a project right now that was designed as microservices from the start. It’s really hard to change design when every change impacts several independent components. It seems to me that microservices are good for very stable and well understood use cases but evolving a design with microservices can be painful.
- fendy3002 6y ago
  Especially if in new requirement, microservice A need to interact with service C which originally accessed via B. You can either: * Make additional changes in B which also takes resources, times and introducing overhead + point of failure, or * Make A interact directly with C which breaks the boundary
majormajor 6y ago
> I would propose alternative is to fix your monolith first. If the team can't rewrite their ball of mud as a new monolith, then what are the chances of successfully rewriting and changing architecture? A million times this. If you can't chart the path to fixing what you have, you don't understand the problem space well enough. The most common reply I've heard to this is "but the old one was in [OLD FRAMEWORK] and we will rewrite in [NEW FRAMEWORK OR LANGUAGE]," blaming the problem not on unclear concepts/hacks or shortcuts taken to patch over fuzzy requirements but on purely "technical" tech debt. But it usually takes wayyyyy longer than they expect to actually finish the rewrite... because of all the surprises from not fully understanding it in the first place. So even if you want the new language or framework, your roadmap isn't complete until you understand the old one well enough.
xwolfi 6y ago
I've noticed after many optimization project in my investment bank that most architectures are fine, and rarely a source of or solution to problems. The source of most of our problems is always that we started too small to understand the implication of our laziness at the start (I'll loop over this thing - oops it's now a nested loop with 1 million elements because a guy 3 years ago made it nested to fix something and the business grew). Most times, we simply have to profile / fix / profile / fix until we reach the sub millisecond. Then we can discuss strategic architecture. Interestingly most of the architecture problem we actually had to solve were because someone 20 years ago chose an event-based micro service architecture that could not scale once we reach millions upon millions of event and has no simple stupid way to query state but to replay in every location the entire stream. Every location means also the C# desktop application 1000 users use. In this case yes, we change the architecture to have basic indexed search somehow with a queriable state rather than a reconstructed one client-side.

by ser0 6y ago

zimpenfish 6y ago

Another issue I've seen is that people push all the problems onto the monolith even if they're external - one place had a Perl monolith (which was bad, sure) but their main issue was an overloaded database which could have been addressed with moving some queries out of the (awful homegrown) ORM and using e.g. signed session cookies instead of every request causing a session table hit.

Chyzwar 6y ago

There is one approach Fowler suggested is SacrificialArchitecture. You build your monolith quickly, get to market fit and once you understand service boundaries you move to microservices. Personally I would like to try Umbrella Projects[1]. You can design it as microservices but deploy and build as monolith. Overhead is lower, and it is easier to figure out right services when in one codebase. It can be easy implemented in other lang/frameworks as well. [1] https://elixirschool.com/en/lessons/advanced/umbrella-projects/ https://elixirschool.com/en/lessons/advanced/umbrella-projec...

andy_ppp 6y ago
This! Service boundaries are vital and almost intractable to design up front as you won’t be sure of your systems’ use cases. I’ve worked on numerous micro services systems and all of them were designed in such a way that data loss and unknown error states were mandatory. Micro services seem simpler, but are actually harder than my methodology “as few services as necessary” + bounded contexts within them. Using network partitions to stop spaghetti code just leads to spaghetti network partitions. The discipline to keep this simple and self contained is the important part.
- radicalbyte 6y ago
  This is why Domain Driven Design should be mandatory reading for all anyone making decisions on Software Architecture.
  - datavirtue 6y ago
    Agreed. But do we mandate that they understand it and somehow enforce their implementation competency? As pervasive and helpful and as easy to understand what Clean Code is, it is still very rare to find someone who understands it and applies it in the field.
  - hackerfromthefu 6y ago
    Unfortunately in actual practise I've encountered significant cargo culting about DDD. Attracts a lot of people with mid level experience who suddenly want to dictate architecture based on theoretical ideals rather than practicalities. There's no substitute for experience, and specifics of adapting architecture to the context of the problem you're trying to solve. In one case I wanted to use a technology that actually matches DDD very significantly - but the cargo cultish closedmindedness of the practitioners meant they couldn't even understand how an old idea/tech they hadn't liked or approved was, was actually an implementation of DDD. The problem there is not DDD, the problem is the people who get closed minded and stuck to their one true way (often without really broad experience to make that judgement call effectively). I've learned that pattern of language of absolutes such as 'should be, mandatory, all' do XXX is often a sign of that kind of cargo cultish thinking.

jtcto 6y ago

I really like how you've worded this and agree. Some of this is driven by well documented cognitive biases, such as the availabily heuristic which you've identified, but there are so many. This is a good video on it https://birgitta.info/redefining-confidence/ https://birgitta.info/redefining-confidence/

tutfbhuf 6y ago

> If the team can't rewrite their ball of mud as a new monolith, then what are the chances of successfully rewriting and changing architecture? Well sometimes there are very complex subsystems in the monolith and it's easier to create a completely new microservices out of that instead of trying to rewrite the existing code in the monolith. We had done so successfully by creating a new payment microservices with stripe integration and then just route every payment that way. Doing the same in a huge pile of perl mess has been assessed as (nearly) impossible by the core developer team without any doubts. But I have to admit that the full monolith code base is in a maintenance mode beyond repair, only bug & security fixes are accepted at this point in time. Feature development is not longer a viable option for this codebase.

pjmlp 6y ago

When the same practices used to sell microservices get applied to modules, packages and libraries, there is no need to put a network in the middle to do the linker's job. But too many are eager to jump into distributed computing without understanding what they are bring into their development workflow and debugging scenarios.

tutfbhuf 6y ago
When I started programming, linking was the only way of packaging software in mainstream computing, if it was so superior we wouldn't have moved away from it.
- pjmlp 6y ago
  Here is the thing, whatever you write doesn't work without linking, so no we never moved away from it.
  - diroussel 6y ago
    But is linking the right way to couple business logic from two different organisational units? That is the question being discussed.
    - tutfbhuf 6y ago
      > But is linking the right way to couple business logic from two different organisational units I think it's not, but I would like to hear other opinions.
      - diroussel 6y ago
        I'm not sure why you are being down voted. In context of calling business logic we have moved away from linking. I guess the context is only implelied in your parent post. The rise of the network API in the last 20 years has proven it's own benefits. Whether you are calling a monolith of a microservice, it's easier to upgrade the logic without recompiling and re-linking all dependencies.

iainmerrick 6y ago

I think the key point of Fowler’s article, which is obscured a little by all the monolith discussions, is that try to start with microservices doesn’t work. He’s claiming this from an empirical point of view -- if you look at successful, working systems, they may use microservices now but they most often started as monoliths. People are talking about Conway’s law, but the more important one here is Gall’s law: “A complex system that works is invariably found to have evolved from a simple system that worked.”

dboreham 6y ago
Conway/Gall duality.

darkstar_16 6y ago

This is the correct way to do things but, unfortunately, like I wrote in my other comment, this knowledge only comes after one has gone through a few of these "transformational" changes. I've seen this happen a few times and monolith vs micro services is such a done deal that it's very difficult to argue against so, even with experience, you will always fail if you try and go against the grain :)

D_Guidi 6y ago

> I would propose alternative is to fix your monolith first I'd like the idea of vertical slices for this: once you've done this, then refactor to microservices will be much easier.

distalx 6y ago

We are just moving complexity from one place to another. And sometimes few things get lost in this transformation.

jimbokun 6y ago

> I would propose alternative is to fix your monolith first. If the team can't rewrite their ball of mud as a new monolith, then what are the chances of successfully rewriting and changing architecture? Often, the problems with the monolith are * different parts of the monolith's functionality need to scale independently, * requirements for different parts of the monolith change with different velocity, * the monolith team is too large and it's becoming difficult to build, test, and deploy everyone's potentially conflicting changes at a regular cadence, with bugs in one part of the code blocking deploying changes to other parts of the code. If you don't have one of these problems, you probably don't need to break off microservices, and just fixing the monolith probably makes sense.

stepbeek 6y ago
> the monolith team is too large and it's becoming difficult to build, test, and deploy everyone's potentially conflicting changes at a regular cadence, with bugs in one part of the code blocking deploying changes to other parts of the code. This is the only argument I'm ever really sold on for an SOA. I wonder if service:engineer is a ratio that could serve as a heuristic for a technical organization's health? I know that there are obvious counter-examples (shopify looks like they're doing just fine), but in other orgs having that ratio be too high or too low could be a warning sign that change is needed.
Mavvie 6y ago
> different parts of the monolith's functionality need to scale independently, I hear this a lot. Can you give me an example? Can't I just add more pods and the "parts" that need more will now have more? Agreed on the other reasons (although ideally the entire monolith should be CI/CD so release cadence is irrelevant, but life isn't perfect)
- jimbokun 6y ago
  So the alternative is to have a monolith deployed with different "identities" based on how it's configured. So you configure and scale out different groups of the monolith code independently, to satisfy different roles and scale up or down as needed. However, sometimes the resources needed to load at start up can be drastically different, leading to different memory requirements. Or different libraries that could conflict and can also impact disk and memory requirements. For a large monolith, this can be significant. So at what point do you go from different "configuration", to where enough has changed it's a truly different service? The dividing line between code and configuration can be very fluid. > the entire monolith should be CI/CD so release cadence is irrelevant But if one module has a bug and is failing testing, or features partially completed, it blocks releasing all the other code in the monolith that is working.

tootie 6y ago

I'm living this right now. A big, fat python 2 monolith maintained for 10 years. Everyone hates it, but it actually works perfectly fine.

spaetzleesser 6y ago

That’s what I am always saying. If you can’t manage libraries you will fail at microservices too, just harder. I am working on a project right now that was designed as microservices from the start. It’s really hard to change design when every change impacts several independent components. It seems to me that microservices are good for very stable and well understood use cases but evolving a design with microservices can be painful.

fendy3002 6y ago
Especially if in new requirement, microservice A need to interact with service C which originally accessed via B. You can either: * Make additional changes in B which also takes resources, times and introducing overhead + point of failure, or * Make A interact directly with C which breaks the boundary

majormajor 6y ago

> I would propose alternative is to fix your monolith first. If the team can't rewrite their ball of mud as a new monolith, then what are the chances of successfully rewriting and changing architecture? A million times this. If you can't chart the path to fixing what you have, you don't understand the problem space well enough. The most common reply I've heard to this is "but the old one was in [OLD FRAMEWORK] and we will rewrite in [NEW FRAMEWORK OR LANGUAGE]," blaming the problem not on unclear concepts/hacks or shortcuts taken to patch over fuzzy requirements but on purely "technical" tech debt. But it usually takes wayyyyy longer than they expect to actually finish the rewrite... because of all the surprises from not fully understanding it in the first place. So even if you want the new language or framework, your roadmap isn't complete until you understand the old one well enough.

xwolfi 6y ago

I've noticed after many optimization project in my investment bank that most architectures are fine, and rarely a source of or solution to problems. The source of most of our problems is always that we started too small to understand the implication of our laziness at the start (I'll loop over this thing - oops it's now a nested loop with 1 million elements because a guy 3 years ago made it nested to fix something and the business grew). Most times, we simply have to profile / fix / profile / fix until we reach the sub millisecond. Then we can discuss strategic architecture. Interestingly most of the architecture problem we actually had to solve were because someone 20 years ago chose an event-based micro service architecture that could not scale once we reach millions upon millions of event and has no simple stupid way to query state but to replay in every location the entire stream. Every location means also the C# desktop application 1000 users use. In this case yes, we change the architecture to have basic indexed search somehow with a queriable state rather than a reconstructed one client-side.