10 ms·
Red Flags Signaling That a Rebuild Will Fail
- lgleason 8y agoI recently left a project that demonstrated most of these traits. Usually these things are the top of the ice-burg.
- teddyh 8y agoKnow your burgs and bergs. A “burg” (or burgh) is a fortification, or more usually refers to a city built around (or inside) that fortification. A “berg” is a mountain, or a large hill. Therefore, an iceberg is an “ice mountain”, and a “burgermeister” is a “city master”; i.e. a mayor.
- draw_down 8y ago:(
- de_watcher 8y agoYou forgot to mention that you should use "tip" instead of "top" in this idiom. Here is a video with more detail: https://www.youtube.com/watch?v=dQw4w9WgXcQ https://www.youtube.com/watch?v=dQw4w9WgXcQ
- alkonaut 8y agoThe truth I think is more often that the legacy system is too old and brittle to improve, and customers are demanding ever more complicated features from it. So you rebuild as a new system as a gamble, because even though it shows all the traits described, the new system is at least one that anyone is willing to develop, and one where features can be added, and to which people can be recruited. We know big rebuilds have small chances of sucess. But that doesn’t mean you shouldn’t do big rewrites. You are in a bad place if you even consider. Maybe the big rewrite means the company has an 80% risk of going under. Still could be that safe bet.
- Chyzwar 8y agoThe rewrite is usually when it is too late for the project. Need for re-write mean that project maintenance was ignored and technical debt reached critical levels. I would start by firing people that led to this situation.
- Buttons840 8y agoThey're already gone, almost certainly.
- lovich 8y agoI've found that they are usually still there but as they are the CEO/CTO it's difficult to get them fired
- nerdponx 8y agoThat, or you are their replacement.
- bumholio 8y agoHey, a bunch of shell scripts glued to some DOS executables were good enough in my time. We had no fancy schmancy github back in those days, yet we built this business on nothing but hard work and pizza. Why, the sources of the DOS exes were long gone by second year, lost in the crash of that old Windows Milenium machine that used to sit in our dorm room and was uniquely configured to compile them using Turbo Pascal - we figured it was a safe option to use as a source repository. But that still didn't stop us - we implemented the remaining features by patching assembly.
- a_imho 8y agoPeople write legacy systems from day 0, especially in resume driven development.
- clintonb 8y agoIf you fire those people, you remove your source of expertise on the old system. Yes, they did a poor job of maintaining the old system, but their knowledge may be valuable to understanding the old system and creating requirements for the new system to reach parity.
- maxxxxx 8y ago"Red Flag #4: You aren’t working with people who were experts in the old system.” I think this is most important. A lot of people want to rewrite because they don't understand the current system and don't want to bother learning. Before you rewrite you really should understand the current state deeply.
- _asummers 8y agoEven having the entirety of the original dev team there, time takes its toll on recollection of reasoning behind some of the strange decisions made in something that would warrant a rewrite. Much preferable to not having them, of course.
- DiabloD3 8y agoSomething I do is if the code looks weird or is rather small for how much work went into it, I leave a comment that says why this was done... just so I can remind myself in 6 months when I go "who the fuck wrote this garbage... oh, me."
- munk-a 8y ago#4 is sort of terribly worded, the summary line is something that is important and pretty independent, make sure you're working with expert users of the system... then the explanation brings in a Senior Dev as a good resource to tap. This is the wrong direction, you really want to consult with the system experts to see their rationale for requesting what might seem like odd functionality in the first place. #4 also mixes a good deal with #5 in that any changes you make (even purely good ones in your view) will require retraining of users and cause a kerfuffle when rolled out to your user base, people _hate_ change.
- pbreit 8y agoThis strikes me as dangerous. Didn't the experts build the first system? Don't you want to deliver a fresher system? Won't the experts be attached to the old way of doing things?
- 8y ago
- pspeter3 8y agoI think people also deeply underestimate the time it will take. We've undergone an incremental rewrite for ~4 years at Asana.
- jupake 8y agoUsed your software once before. Loved it! You guys should do a blog post about your rewrite experience. Would love to know what your tech stack was and what your new one looks like.
- solox3 8y agoWith this good article I think I have a good question. The reference to Martin Fowler’s strangler pattern (https://www.martinfowler.com/bliki/StranglerApplication.html https://www.martinfowler.com/bliki/StranglerApplication.html) was mentioned in the article to grow the new system in the same codebase until the old system is strangled. In my case (Ionic 1 to 2) however, both the entire framework and the language are different. How should the strangler pattern work in this case?
- ronpeled 8y agoThere's no super easy way here. One way to get this done is find independent areas of the app that can be replaced without coupling. Then start building up as you go with the new system. At some point you'll be about 70% through of which you can decide if you want to make the jump and focus your efforts to completely uproot the old one. Sorry for the abstract reference here, but it applies to almost any replatforming out there. In most cases it is a very expensive operation for a business and needs some major reasons in order to justify such a move.
- omegaworks 8y agoAre there any examples out there for how to do this with React in an existing AngularJS codebase?
- gknoy 8y agoInstead of a Single Page Application, make it an MPA (multi-page application), each of which is basically a separate SPA. You get latency when swapping between sections of your app, but on some codebases (such as for an internally-used app), that's less of a problem. We did something similar to this when we broke up our Ember application so that we could code new things in React. We still maintain our Ember codebase, but are rewriting parts of some routes in React, and adding all new things in the React app. We deploy ours as separate pods in a Kubernetes cluster, but you could even host them on the same server with separate nginx routes. The initial ramp up of this is a little frustrating, as it seems you're adding extra overhead to everything, the long term goal is to have infrastructure and workflow that supports having part of your app in The Old Proven Thing, and part in The New Hotness. This is valuable whether you're switching to React, or upgrading from Ember 2 to 3, etc, as it lets you upgrade a smaller set of dependencies, and experiment with things.
- nerdponx 8y agoAnother red flag not mentioned here: the old system doesn't have an end-to-end suite of functional test cases you can rely on.
- Ensorceled 8y agoRed Flag #6: Key stake holders keep moving the goal posts. If your goal moves from feature comparable but on a modern platform, to new features, to a complete reinventing of the product all without actually shipping ... you might be in trouble. I had a rebuild go 6 months over. In the heated executive meeting at t+3 months I was called to defend my team and pointed out that the VP Product had just delivered “final” specs literally the day before. How could we be on track with development if PM is 3 months past “end of development” with design specifications. The fact that the specs were changing weekly because “we’re agile” is a whole other issue.
- whatshisface 8y agoPeople sometimes complain about how developers like to "write the operating system and then a language" when it comes to handling every foreseeable permutation of what the program might every be desired to do, but we're all so used to unstable requirements that sometimes the metaphorical programming language research is the only thing that will be general enough to find a use next week.
- oneplane 8y agoI almost had a few similar situations, but after pointing out that being agile doesn't just means changing requirements but also changing time paths or simply different deliveries after each change it got a whole lot clear what agile (and scrum) is good for, and what it's not good for (i.e. agile process but expecting waterfall results doesn't work).
- Cthulhu_ 8y ago> The fact that the specs were changing weekly because “we’re agile” is a whole other issue. The article touches on that too; simplified it's stating that if you're not live within 6 months, you're doing waterfall.
- Ensorceled 8y agoThat’s not waterfall. Waterfall you don’t start dev before specs are final. Waterfall isn’t just a synonym for “the wrong way to do it” :-)
- nostrademons 8y ago#5 has a converse - oftentimes, the only way to get a rebuild to succeed is to drop features, and it's a major red flag if management insists on 100% feature parity. The way to distinguish this from the #5 situation in the article is to ask if you're dropping features because they're hard or because nobody uses them. The former is a red flag; the latter is a green flag. Before you embark on a rebuild, you should have solid data (ideally backed up by logs) about which features your users are using, which ones they care about, which ones are "nice to haves", which ones were very necessary to get to the stage you're at now but have lost their importance in the current business environment, and which ones were outright mistakes. And you should be able to identify at least half a dozen features in the last 3 categories that you can commit to cutting. Otherwise it's likely that the rewrite will contain all the complexity of the original system, but without the institutional knowledge built up on how to manage that complexity.
- maxxxxx 8y agoI have seen that before. You kill yourself refactoring a feature only to find out it's never or barely used. Deleted code and features are the best.
- hinkley 8y agoYou hamstring the product to make a feature work one way then find out that what they really wanted would have been easier to implement but they never asked because they thought that would be harder.
- watwut 8y agoThat is why social skills and good analysts are important.
- kwhitefoot 8y agoI've been fighting this problem for my whole software development career (more than 35 years now and no end in sight).
- tomelders 8y agoI’ve carved a career out of rebuilds. I’m working on a rebuild right now. There’s a ton of companies out there who’ve done very well with their home grown antiquated systems from the late 90’s and early 00’s that are now facing stiff competition from young upstarts who had feature parity from day one and are knocking out new features at break neck pace because they’re leveraging the latest and greatest in tools, technology, and thinking. I’ve always been a big believer in rebuilding your product from the ground up. I think it’s something you should always have going on in the background. Just a couple of devs whose job it is to try and rebuild your thing from scratch. Maybe you’ll never use the new version. But I think it’s a great way to better understand your product and make sure there’s no dark corners that no one dare touch because they don’t understand what it does, how it does it, or why it does it the way it does. And I’ve always believed that if you don’t want to rebuild your app from scratch, then don’t worry, a competitor will do it for you. So I agree with every point raised in this article. And I think it does a great job of articulating the issues that often go unspoken. But I’d like to add one more. And for me, this is the biggest issue for any company wanting to rebuild it’s product. If your sales team has more clout than your designers and developers, then you’re fucked. And in the enterprise software world, this is the norm. An uncheked sales team that get’s whatever it wants has already killed your product and made it impossible to rebuild. Their demands are ad-hoc, nonsensical, and always urgent. So urgent that proper testing and documentation are not valid reasons to prevent a release. Their demands are driven by their sales targets, and the promises they make to clients are born out of ignorance of what what your product does, and how it does it. This is not true of all companies. Many companies find a reasonable balance between the insatiable demands of a sales force and the weary cautiousness of their engineers. But if your company submits to every wish and whim of your sales team, and you attempt to rebuild your product, then you’re screwed.
- omeid2 8y ago> I’ve always been a big believer in rebuilding your product from the ground up. I think it’s something you should always have going on in the background. Just a couple of devs whose job it is to try and rebuild your thing from scratch. Hahaha. Just a couple of devs?
- wellpast 8y agoRed Flag #1 should be that you’re doing a rebuild.
- CydeWeys 8y agoI'm potentially looking at a situation like this right now at work. We're on a NoSQL DB and it's just not working too well for us anymore, so we would like to transition to something that provides more relational semantics (PostGres, Spanner, something like that). Migrating the backend between one kind of DB and another is non-trivial, especially because the whole ORM needs to be ripped out as well. It's not a full rebuild of the application but it's definitely substantial in effort level. Sometimes a rebuild is just necessary, because you are on a tech stack that is no longer working for you, for whatever reason. How would you solve that kind of problem?
- wellpast 8y agoAs incrementally as possible. Eg, does your entire data model need to move at once? (Probably not.)
- grey-area 8y agoI'd definitely vote for PostgreSQL, it can handle large loads effortlessly, it's reliable, and yet they keep adding great features. It could also function pretty much like a nosql db initially, to ease your transition, then you could migrate gradually to using it as a relational db. You need strong checks on data integrity before you start - you could consider double writing (to old orm using nosql + new orm using psql), and comparing data stored to be sure you don't miss anything at first, before you switch?
- hvidgaard 8y agoThe first thing you do is refactor with the existing DB, so you have a clear DataStore component. Then you make your shiny Relational DB implementation of that DataStore. Now you run both side by side and for everything you do in the old DB you do the same in the new DB, and you compare the results. At some point you can turn off the old DB with confidence and sleep well knowing that the new DB behaves the way you expect.
- lyqwyd 8y agoThis article really captures the risks of a rebuild. I’ve been through a number of them, all but 1 abject failures. The one success was driven by the executive understanding that the company would fail without a rebounds, and it was still 6 months late, resulted in one of the cofounders being fired, an extremely painful rollout, and the company still failed, due to other problems. My firm belief is that when you need a rebuild, you are already well into a fail state as a company. Not to stay there can be no recovery, but it is an indication of some deep problems for the company, beyond anything the engineering department alone can resolve... and if the rebuild is not coming from the executive leadership, it is an even bigger issue as it will more likely lead to bigger problems than it will solve.
- kazishariar 8y ago¯\_(ツ)_/¯:'Dual commits' to the rescue! -pun intended
- gaius 8y agoMissing the biggest red flag of all, engineers wanting to just play with new toys and pad their CVs. Ask the engineers why they want to rebuild and listen carefully to the answer and if it’s vague handwaving and buzzwords (microservices! Containers! New JS framework!) and no hard numbers to justify it, just say no. For example “we spend X/year on AWS but if we spend Y to rewrite in C++ we need fewer VMs and can cut that to Z/year” is simple calculations. If your engineers can’t even do that, their motives are suspect.
- shoo 8y agohttp://boringtechnology.club http://boringtechnology.club
- adrianN 8y agoThe problem is that Y and Z are just numbers you make up. Reliably estimating them is impossible without at least building a prototype.
- ebiester 8y agoOn the other hand, “we cannot hire anyone to work in COBOL/Perl 5.8/Tcl/other outdated language” is a very real problem. It turns out that 2018, developers are judged for working too long in old technologies even when we know as in industry that a developer can learn a new language.
- 8y ago
- rwmj 8y agoIs "rebuild" new jargon for "rewrite", or does it mean something different? I thought the article was going to be about builds failing.
- ConceptJunkie 8y agoYeah, I did too until I started reading the article. Using the normal sense of "rebuild" didn't make sense.
- ellimilial 8y agoThis is gold. I've become a member of a team the company scrambled to deal with a `legacy` python/SQL - based ingestion/storage system in an effort to 'harden' it. Despite my best efforts, we are going for a full rewrite into java/spring/avro/mongo/es. We have internal users talking SQL and utilising the system at the moment, a fair amount of data is relational. I have run out of ideas how to convince the team and stakeholders, will have a one-shot chance to talk to VP. Any ideas how to voice the concerns about the full re-design (perhaps I'm just being difficult)?
- sonnyblarney 8y ago1. Given the risk, cost and limited upside, the onus is on the refactor team to prove that it needs to be done. Where is the ROI, factor in the risk. Where is this in the stack of things to do? Are there better ROI things? 2. Consider 'what the point' is in the first place, because the entire world could be run on python/SQL and it would be 'hard'. I don't think anyone would consider 'Mongo' to be 'hard' usually people use it because it's fast and easy, not hard. Consider maybe only replacing one part at a time, i.e. Java-SQL. 3. Consider a simple clean up or refactor. No need to learn no languages and tools when maybe you just need a house clean. 4. People seem to be going back to SQL because of it's inherent standardization - so many reporting and analysis systems use SQL as an interface, to the point where even NoSQLs are starting to use SQL.
- pedalpete 8y agoI'm a big supporter of "replacing one part at a time", and wish I had done that on a rebuild I'm just completing. In fact, I thought I was. We split our app into 3 parts, rebuilt part 1, then part 2, but part 1 couldn't be released to customers until part 2 was done, and we kept our legacy system supporting the majority of our users until we are done with part 3, which is nearing completion now. I thought that was "replacing one piece at a time", but it isn't most users aren't touching it until part 3 is done, and at that point, they are experiencing a new system from scratch.
- mratzloff 8y agoWithout knowing the performance requirements and where the current system is failing, it's hard to know if the technology stack will work for your needs—with one exception. If users speak SQL, they will reject Mongo. The users of the system are the ones who will determine project success or failure. Think about the data analysts, product owners, etc. who use the system. Interview them. Find out exactly how they use the system currently. Do they query in an ad hoc way? Do they rapidly iterate on their queries? Watch them interact with the system. If it's any way other than through dashboards that an engineer updates on request, you are in for rough seas. Users must always determine the contours of a new system. There are big data solutions that speak SQL. Some are cloud-based, some are not. Some are faster than others. The team should be able to show you why they rejected those as solutions.
- bkovacev 8y agoThis is yet another article where there's a clear managerial-only approach. Sorry, but I dont dig this. As a developer you're constantly fighting managers who want to rush things to get them out and who will eventually blame you for a bug/non-defined behavior once you hit a certain milestone. To me it seems the author of the article doesn't understand the tech debt. If you've ever worked in a startup you'd know that the requirements are ever-changing, thus that if a certain payment system is put in place, it might evolve to the point where you really need to refactor it and in order to enable the refactor you have to refactor the whole business flow as well. If there's more than 2-3 features affected by a new feature, a big refactor is definitely needed. Only one solution offered, which I dont think is adequate because why would I leave something in that was only meant to provide value for short term and then build on top of it till I kill the old system?
- LolNoGenerics 8y agoHis argument is against rewriting a whole codebase. Refactoring is surely an alternative.
- jpeeler 8y agoFirefox seems to be doing pretty well with their incremental rewrite into rust. I do wonder how long it will take to complete the transition versus doing a complete rewrite instead.