10 ms·

Maybe getting rid of your QA team was bad

433 points by nlavezzo 3y ago

deleted 3y ago
[deleted]
fatnoah 3y ago
In making the case for building up a QA org at my current startup, I repeat the mantra that QA is both a skillset and a mindset. Automated tests can tell us a lot, but skilled QA testers are amazing at edge cases to break things and providing human feedback about what looks and feels good gor users.
robotnikman 3y ago
Microsoft is probably the one case where this sticks out the most, at least for me anyways. Noticeably more bugs in updates since they dropped their QA team, in Windows as well as cloud products.
- EvanAnderson 3y ago
  The revenue keeps rolling in so, clearly, they made the right business decision... >sigh< I use a lot of MSFT software and services in the "day job". I wish there was some kind of consequence to them for their declining quality.
  - danesparza 3y ago
    Almost like you should be able to file a 'bug report' or something. Maybe they should build a team to make sure their code quality is up to snuff...
  - shiroiuma 3y ago
    >The revenue keeps rolling in so, clearly, they made the right business decision... >sigh< This is exactly the point I was going to make. Their stock price is doing great! So obviously they've done the right thing for their position in the market: the market has rewarded them for not wasting money on QA and just letting users suffer with the bugs. >I wish there was some kind of consequence to them for their declining quality. If people keep insisting on throwing money at them no matter how bad their software is, then there's no reason for them to improve their quality.
alanjay 3y ago
But Microsoft fired their entire qa department, and their software is rock solid. Right! Right?
- deleted 3y ago
  [deleted]
mschuster91 3y ago
Another part is that there is barely any training for QA people. Even your average CS course will only graze the top of the topic, most usually some prof droning on about Java unit tests on some really old version of Java and a testing framework just as old. There are no "Software Quality Assurance" academic degrees, there's barely any research into testing methodologies, there's barely any commercial engagement in the space aside from test run environments (aka, selling shovels to gold diggers), and let's face truth, also in tooling. And everything but software QA is an even worse state, with "training" usually consisting of a few weeks of "learning on the job". Basic stuff like "how does one even write a test plan", "how does one keep track of bugs", "how to determine what needs to be tested in what way (unit, integration, e2e)" is at best cargo-culted in the organization, at worst everyone is left to reinvent the wheel themselves, and you end up with 13 different "testing" jobs in a manually clicked-together Jenkins server, for one project. > Defect Investigation: Reproduction, or “repro”, is a critical part of managing bugs. In order to expedite fixes, somebody has to do the legwork to translate “I tried to buy a movie ticket and it didn’t work” into “character encoding issues broke the purchase flow for a customer with a non-English character in their name”. And this would normally not be the job of a QA person, that's 1st level support's job, but outsourcing to Indian body shops or outright AI chatbots is cheaper than hiring competent support staff. That also ties in to another aspect I found lacking in the article: users are expected to be your testers for free aka you sell bananaware. No matter if it's AAA games, computer OSes, phones, even cars...
- wmichelin 3y ago
  To play devil's advocate here, I did not _really_ get training for my software engineering role. I got a little bit from 1-2 college courses, but the vast majority of my role I had to pick up on the job or on my own. I can tell you, I definitely didn't get training for QA tasks, but here I am doing them anyways. It's just work that needs to be done.
  - mschuster91 3y ago
    > To play devil's advocate here, I did not _really_ get training for my software engineering role Yeah and that is my point. It would be way better for the entire field of QA if there were at least a commonly agreed base framework of concepts and ways to do and especially to name things, if alone because the lack of standardization wrecks one's ability to even get testers and makes onboarding them to your "in house standards" a very costly endeavour.
wmichelin 3y ago
This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering. I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests". With this model, I have found that the engineering team ends up being the QA team in practice, and then the actual QA team often only finds bugs that aren't really bugs, just creating noise and taking away more value than they provide. I would love to learn about QA team models that work. Manual tests are great, but they only go so far in my experience. I'm not trying to knock on QA folks, I'm just sharing my experience.
- convolvatron 3y ago
  in the classic model, most QA orgs were a useless appendage. partially by construction, but largely because QA gets squeezed out when dev is late (when does that happen?). they aren't folded in early, so they twiddle their thumbs doing 'test infrastructure' and 'test plans', until they finally get a code drop and a 48 hr schedule to sign off, which they are under extreme pressure to do. but every once and a while you ran across a QA organization that actually had a deep understanding of the problem domain, and actually helped drive development. right there alongside dev the entire way. not only did they improve quality, but they actually saved everyone time.
  - lambic 3y ago
    Not sure why this was downvoted, that second paragraph is right on the money.
  - cratermoon 3y ago
    Saying "useless appendage" sounds to me like it's the QA team that's the problem, when what you're really saying is that it's the organization and process that pushed QA teams into irrelevance. I agree with your assessment overall, and those issues were one of the driving forces behind companies dispensing with QA and putting it all on the developers.
  - 3y ago
cloths 3y ago
I can't agree more. > Focus: There is real value in having people at your company whose focus is on the quality of your end product. Quality might be “everybody’s job”…but it should also be “somebody’s job”. Yes indeed, naturally every person have just one focus, having dedicated person focus on QA is important. Another practice, or buzz word (or used to be buzz word:) ), Exploratory Testing, which can pretty much be conducted only by dedicated QA.
- philk10 3y ago
  That's pretty much my role - I don't write test cases, I'll explore the system and try to find issues that the devs have missed. Then they learn from what they missed so I have to explore more to find other types of issues.
gyudin 3y ago
Getting rid of QA teams are a slow ticking bomb frequently imho. Cause some issues are might not be even breaking functionality. You can mess up some tracking/analytics and managers will make wrong decisions based on incorrect data. But personally I feel like within few years everything might change a lot. Machines 100% will be be better at coding, maintenance and testing things.
harshalizee 3y ago
This is a symptom of a larger issue in tech where C/E suites are trying really hard to turn engineers into some sort of fungible cogs in the system that can be swapped in and out and in different parts of the system and still have everything work perfectly in order.
- righthand 3y ago
  Yep I see this a lot where people want swappable engineers and no one is able to understand if you have an engineer working in the Frontend most of the time they will not be acquainted with backend work. Nor is there a need or logical way to keep everyone working across the stack to keep them in this swappable state. Each time you change a persons job they need retraining and orientation. Code is code but a drop down menu is not a database insert.
righthand 3y ago
QA Engineers are some of the best debuggers too. They have their hands in the pipeline, src and test directories, and often work with all aspects of developing and deploying the application. When I was a QA lead I often ran into software engineers that couldn’t be bothered to read a pipeline error message (and would complain daily in Slack) and when it came to optimizing the pipeline they would ignore base problems and pretend the issues that stemmed from the base problems were magical and not understood. Wasting days guessing at a solution. The disrespect a QA engineer sees is not exaggerated in this article. Since most companies with QA orgs do not have a rigorous interviewing process like the Engineering orgs, the QA engineers are seen as lesser. The only SWE that have respect for them that I’ve met are the people who worked in QA themselves. The disrespect is so rampant that I myself have switched back to the Engineering org (I tried using seniority as a principal engineer and even shifted as a manager to make changes, but this failed because Engineering could not see past their own hubris and leadership peoples will not help you). My previous company before I was laid off hired a new CTO who claimed we could just automate away QA needs but had no examples of what she was talking about. This is the level of respect poured down from the top about building good software.
- deleted 3y ago
  [deleted]
ptmcc 3y ago
I stepping-stoned through QA on my way into development, now a decade something ago, and this part stands out as especially true in my experience: > This created a self-reinforcing spiral, in which anyone “good enough at coding” or fed up with being treated poorly would leave QA. Similarly, others would assume anyone in QA wasn’t “good enough” to exit the discipline. No one recommends the field to new grads. Eventually, the whole thing seemed like it wasn’t worth it any more. Our divorce with QA was a cold one — companies just said “we’re no longer going to have that function, figure it out.” I've worked with a handful of talented career software QA people in the past. The sanity they can bring to a complex system is amazing, but it seems like a shrinking niche. The writing was on the wall for me that I needed and wanted to get into fulltime dev as QA got increasingly squeezed, mistreated, and misunderstood. At so many companies QA roles went into a death spiral and haven't recovered. Now, as the author points out, a lot of younger engineers have never worked with a QA team in their career. Or maybe have worked with a crappy QA team because its been devalued so much. So many people have never seen good QA that no one advocates for it.
- reactordev 3y ago
  >anyone in QA wasn’t “good enough” This is why. Engineers have some of the most inflated egos that they set an extremely high bar for being “part of the club”. Sometimes that’s corporate policy (hire better than you) and sometimes it’s just toxicity (I am better than you). Without realizing that the most valuable skills they could learn are soft skills. I’m open to finding anyone willing to code. Whether it’s from QA, sales, Gerry’s nephew, recent CS grad, Designer turned coder, or that business analyst that taught themselves Python/Pandas. A good QA team is sorely missed. A bad QA team turns the whole notion of QA teams sour. Just the same for development teams :D I think devs are first line of defense. Unit tests etc. QA is second line (should we release?), feature testing, regression, UX continuity, etc. There’s value in it if you can afford it.
annoyingnoob 3y ago
I remember a time before agile and devops. Seems like QA has always been looked down on, and always considered a bottleneck.
oaththrowaway 3y ago
I had a boss at Yahoo who gave our QA to another team because "Facebook doesn't use QA, we shouldn't either". I can't remember if it was Facebook or MS, but he was willing to buy all of us a book talking about how amazing it was. Long story short, it wasn't. It was like taking away a crutch. Of course we could have been more diligent about testing before having QA validate it, but it slowed development down so much trying to learn all the things we never thought to test that QA did automatically.
- robocat 3y ago
  An article about Facebook's reason for no QA with some of the mitigations: https://blog.southparkcommons.com/move-fast-or-die/ https://blog.southparkcommons.com/move-fast-or-die/ A bit recent to have affected Yahoo - but it sells a good story. We would celebrate the first time someone broke something. Let anyone touch any part of the codebase and get in there to fix a bug or build a feature. Yes, this can cause bugs. Yes, that is an acceptable tradeoff. We had a shared mythology about the times that the site went down and we all rallied together to quickly bring it back up. Sounds like hell: running as close to the edge of the cliff as you can. Presumably totally ignoring thousands of papercuts of slightly broken functionality. Optimising to produce an infinite number of shallow bugs.
  - quadrifoliate 3y ago
    The difference is that bugs in the social network parts of Facebook (the ones where you see your friends and family's pictures and posts) are not directly making money for Facebook. The only real stuff that matters for the money is all the tracking. I bet the people responsible for Facebook Ads Manager are a lot less enthusiastic about "move fast and break things", although I'd be interested to hear an opposing viewpoint from anyone here who's worked for that group.
amlozano 3y ago
This point is brought up in the article but I think it is at the real heart of the issue. QA is almost always seen as a 'cost center' by the business and upper management. I have a hypothesis that you never ought to work in a department that is seen as a 'cost center'. The bonuses, the recognition, and the respect always goes to the money makers. The cost center is the first place to get more work with less hands, get blamed for failures, and ultimately fired when the business needs to slim up. I think the same thing applies to IT. This spiral is why QA will always be a harder career than just taking similar skills and being a developer. It self reinforces that the best people get fed up and switch out as soon as they can.
- robofanatic 3y ago
  > QA is almost always seen as a 'cost center' by the business and upper management Well everything involved in making a product is seen as a cost, that includes the entire development team - QA, Developers, Devops, PM ....
  - rubidium 3y ago
    No. That’s not actually how most orgs break it down. R&D, marketing, sales is “bringing new business” so are profit centers. This means their budget grows with revenue. Manufacturing, QA, IT and service are cost centers so get squeezed year-over-year even if revenue is flat.
    - itsdrewmiller 3y ago
      What software companies are not classifying QA as R&D?
    - jdlshore 3y ago
      It depends on the org. In my company, which is a SaaS-like company, all of product and engineering is a cost center, despite creating the product the company sells. It’s just the way they do their accounting.
- serial_dev 3y ago
hasoleju 3y ago
I completely agree with the sentiment of this article: It is a big problem that being a "software tester" is not at all as prestigious as being a software engineer. Having someone who really understands how the users interact with the software and systematically covers all behavior in test cases is very valuable. I experienced both worlds: I worked in an organization where 4 QA engineers tested each release that was built by 6 software engineers. Now I'm in a situation where 0 QA engineers test the work of 8 software engineers. In the second case the software engineers actually do all the testing, but not that systematically because it's not their job to optimize the testing process. Having someone with the capabilities of a software engineer who's daily work is uncovering defects and verifying functionality is important. Paying someone who owns the testing process is more than justified commercially. The problem is: You don't find those people. For various reasons. Therefor you are stuck with making the software engineers do the testing. But there is hope. There is a new standard released for my industry that requires organizations to have a QA department that is independent of the software engineering department. If they don't have that, they are not allowed to role out there software product complaint to the standard. Maybe this will help to reintroduce the QA Engineer as an important and prestigious role.
- zabzonk 3y ago
  what industry is that?
  - hasoleju 3y ago
    Mechanical engineering and shopfloor software for factory automation in Europe. There is a new IT security standard released that also includes requirements for the software.
godelski 3y ago
The main problem with QA teams is the same problem with IT teams or even management. If they are doing their jobs well they appear to be doing nothing. This often creates a situation where people need to "justify" their jobs. Usually this happens due to an over reliance upon metrics (see Goodhart's Law) rather than understanding what the metrics are proxying and what the actual purpose of the job is. A bad QA team is one who is overly nitpicky, looking to ensure they have something to say. A good QA team simultaneous checks for quality as well as trains employees to produce higher quality. I do feel like there is a lack of training going on in the workforce. We had the "95%-ile isn't that good"[0] post on the front page not long ago and literally it is saying "It's easy to get to the top 5% of performers in any field because most performers don't actively train or have active feedback." It's like the difference between being on an amateur sports team vs a professional. Shouldn't businesses be operating like the latter? Constantly training? Should make hiring be viewed differently too, as in "can we turn this person into a top performer" rather than "are they already" because the latter isn't as meaningful as it appears when your environment is vastly different than the one where success was demonstrated. [0] https://news.ycombinator.com/item?id=38560345 https://news.ycombinator.com/item?id=38560345
- philk10 3y ago
  When I work with new devs I can often trip them up with basic tests of double-clicking, leading spaces, using the back button on Android. They then learn these and from then on these issues dont appear ( well, OK, it might take a couple of times of a ticket being rejected because of these but they do quickly learn all my tricks ) I don't get measured on bugs found so there's no pressure on me to find stupid bugs just to boost my figures.
  - hotpotamus 3y ago
    I had a buddy who liked to do that kind of thing. I think his favorite trick was just to enter nothing in a form and hit enter to see what happens. It's probably his favorite because it deleted the database on some little thing I wrote at one point and we got a good laugh over it and I got a good lesson out of it.
itqwertz 3y ago
I noticed this trend a couple of years ago, they called it Shift-Left. Basically, it was a way to get a developer to do more testing during development before handing over a feature to QA. This sets up the imagined possibility of firing all QA staff and having developers write perfect code that never needs to be tested thoroughly. Looks great on paper... At a previous company, they started firing all of the manual QA devs and replacing them with offshore QA people who could do automated testing development (Cypress tests). The only problem was that those fired QA team members had significant business domain knowledge that never was transferred or written down. The result was a dumpster fire every week, with the usual offshore nightmares and panicked responses. Make no mistake about this, it's just a cost-cutting measure with an impact that is rarely felt immediately by management. I've worked with stellar manual QA people and our effort resulted in bulletproof software that handled edge cases, legacy customers, API versions, etc. Without these people, your software eventually becomes IT-IS-WHAT-IT-IS and your position is always threatened due to the lack of quality controls.
deleted 3y ago
[deleted]
evilantnie 3y ago
QA has always been about risk management. There are multiple ways to manage risk, and some of those ways can be more cost effective to a business. As software shifted towards SaaS offerings, deployments (and rollbacks) became quicker, customer feedback loops also got lightning fast. Team's can manage the risk of a bug more efficiently by optimizing for mean-time-to-recovery. This muscle is not one that QA teams are particularly optimized for, thus their effectiveness in this new model was reduced. I've found that holding on to QA function in this environment can severely dilute the ownership of quality as a requirement from engineers. QA is still extremely valuable in any software that has long deployment lead times. Mobile apps, On-Prem solutions, anything that cannot be deployed or rolled back within minutes can benefit from a dedicated QA team that can manage the risk appropriately.
- spinningD20 3y ago
  There are so so many instances where "rolling back" is just not a feasible solution. Working for a SaaS company with mobile/web/api and huge db's, migrations, payroll uses in the product, rolling back is and should always be a LAST RESORT. In 99% of the cases, something significant enough to want to roll back usually results in a "hot patch" workflow instead because rolling back or etc has its own risk. > QA has always been about risk management. 100%. QA should be related to identifying risk, likelihood of failure, impact of failure to user, client and company. The earlier this is done in the varying processes, the better. ("shift left" but I've seen a ton of differences with how people describe this, but generally QA should start getting involved in the "design phase") Another example from my own first-hand experience: A company I worked for made a product that plugged into machines that were manufacturing parts, and based on your parameters it would tell you whether or not the part was "good" or "bad". When interviewing the leadership of the company, as well as the majority of the engineering group, "what is the biggest risk with this product" they all said "if the product locks up!". Upon further discussion, I pulled out a much larger, insidious risk; "what if our product tells the client that the part is 'good' when it is not?" In this example, the part could be involved in a medical device that keeps someone alive. You're not going to be able to roll that back.
blastbking 3y ago
Agree with the sentiment of this article but the disturbing ai generated images every paragraph were definitely not necessary - do people actually need to see these?
- jawns 3y ago
  It did strike me as ironic that the article is about ruthlessly automating to avoid paying QA engineers, and it uses AI to avoid paying illustrators.
  - willsmith72 3y ago
    not saying this is you, but i get so tired of feedback about ai-generated images along the lines of "you're taking money away from local artists" it's not one or the other. in my experience it's a decision of "no images" vs "ai images". in this case, probably "no images" would've been better for the reading experience. but there was never any illustrator getting paid
- Night_Thastus 3y ago
  I'd rather have stock or existing images, or none at all. Every time I looked at one of those AI images my brain just kept seeing all the little weird parts that didn't make sense. Like a brain itch.
somewhereoutth 3y ago
Systems used by humans have to be tested by humans. Those testers can either be your customers, or your QA team - as your dev and sales teams will be busy doing other things.
amtamt 3y ago
> The most conscientious employees in your organization are the most bitter. They see the quality issues, they often address them, and they get no recognition for doing so. When they speak up about quality concerns, they get treated like mouthbreathers who want to slow down. They watch the “move fast and break things” crowd get rewarded time after time, while they run around angrily cleaning up their messes. To these folks, it feels like giving a damn is a huge career liability in your organization. Because it is. This is the bitter truth, no one wants to acknowledge. DBAs and Infra, are in the same boat as QAs. Pendulam will swing back in not so long time frame i hope.
- jabroni_salad 3y ago
  My idea of the pendulum swinging back is 'you build it you run it', personally. Don't like the oncall pager? don't make it ring.
  - hinkley 3y ago
    The hard part for most people is learning to tell the devil in a necktie to fuck off every time they try to sweet talk you into volunteering for that kind of pain. Also with no recognition or compensation. All it takes is one person on my team to defect and support an untenable amount of tech debt, and everyone on my team has to pay for it.
- esafak 3y ago
  No, just jump ship and let the damn company fail. Fail faster, haha! This is how we have nice things; when bad companies are not propped up.
- sporedro 3y ago
  While QA and testing is important, I’m not sure you can convince the people only concerned about profits… I think the “release it fast and patch it later” concept here to stay due to the internet being so accessible. Why bother spending tons of money and time when the users will just report the bugs and you can release updates over the internet they can download. Ever since physical copies of video games and software were replaced mainly by downloads, it seems like patching is cheaper. Of course this leads to horrendous security issues, bad user experience, etc. but who cares as long as the guy on top is maximizing profits.
karmakaze 3y ago
Having a QA team is like having an Ops team with stuff being 'thrown over the wall' to the downstream. There's two kinds of tests. Regression testing, that should be automated and written and maintained by devs. New feature or change testing should be done by those that defined them, namely Product people. In the best case it's an iterative and collaborative process, where things can be evaluated in dev/test environments, staging environments, or production for beta flag enabled test users.
Zelphyr 3y ago
I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had final say whether the release went to production or not. These days companies think having the developers write automated tests and spend an inordinate amount of time worrying over code coverage is better. I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. I'm not saying automated testing is bad. I'm saying, just as the author does, that doing away with human QA testers is.
- supportengineer 3y ago
  I've seen QA/QE greatness and it was similar to how you describe. A different chain of command for deciding if releases are certified for production. Different incentive structures as well. Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.
  - eitally 3y ago
    I had a team like that once. It was glorious. And ultimately, I'm convinced it led to overall faster development cycles because the baseline code quality & documentation was so much better than it would have been without such a great QA manager. The QA team, of course, was also technical -- mostly with SWE backgrounds -- and they were primarily colo'd in the same office as the dev team. I still remember the epiphany everyone had one planning cycle when it was mutually understood that by generally agreeing to use TDD, the QA team could participate actively in the real engineering planning and product development process. ... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".
RedShift1 3y ago
Now just to convince msft of this fact because their shit's been breaking left and right and all over the place.
temuze 3y ago
I strongly disagree. I worked at a company with a world-class QA team. They were amazing and I can't say enough nice things about them. They were comprehensive and professional and amazing human beings. They had great attention to detail and they catalogued a huge spreadsheet of manual things to test. Engineers loved working with them. However -- the end result was that engineers got lazy. They were throwing code over to QA while barely testing code themselves. They were entirely reliant on manual QA, so every release bounced back and forth several times before release. Sometimes, we had feature branches being tested for months, creating HUGE merge conflicts. Of course, management noticed this was inefficient, so they formed another team dedicated to automated QA. But their coverage was always tiny, and they didn't have resources to cover every release, so everyone wanted to continue using manual QA for CYA purposes. When I started my own company, I hired some of my old engineering coworkers. I decided to not hire QA at all, which was controversial because we _loved_ our old QA team. However, the end result was that we were much faster. 1. It forced us to invest heavily on automation (parallelizing the bejesus out of everything, so it runs in <15min), making us much faster 2. Engineers had a _lot_ of motivation to test things well themselves because there was no CYA culture. They couldn't throw things over a wall and wash their hands of any accountability. We also didn't have a lack of end-to-end tests, as the author alludes to. Almost all of our tests were functional / integration tests, that run on top of a docker-compose set up that simulated production pretty well. After all, are unit tests where you mock every data source helpful at all? We invested a lot of time in making realistic fixtures. Sure, we released some small bugs. But we never had huge, show stopping bugs because engineers acted as owners, carefully testing the worst-case scenarios themselves. The only downside was that we were slower to catch subtle, not-caught-by-Sentry bugs, so things like UX transition weirdness. But that was mostly okay. Now, there is still a use case for manual QA -- it's a question of risk tolerance. However, most applications don't fit that category.
- mixmastamyk 3y ago
  False dichotomy. Poor dev practice is not fixed by elimination of QA, but rather fixed by improving dev practice. The “five why’s” can help.
  - 3y ago
Ensorceled 3y ago
I hired a great QA Developer a few months ago. They are building out integration, performance and end to end tests; finding bugs, speeding up releases and generally making things better. I get asked, every week, if they are contributing and what are they contributing. It's exhausting, so I can't imagine what it feels like to actually BE in QA.
- warkdarrior 3y ago
  # of bugs found per week should be sufficient metric of productivity.
bgribble 3y ago
I was lucky enough to work in a small eng team with 1 full-time dedicated QA person. One of the very few coworkers from my long career that I have really tried hard to poach away from whatever they were doing after our shared workplace went bust. Yes, part of the job was to write and run manual test suites, and to sign off as the DRI that a certain version had had all the automated and manual tests pass before release. But their main value was in the completely vague mandate "get in there and try to break it." Having someone who knows the system and can really dig into the weak spots to find problems that devs will try to handwave away ("just one report of the problem? probably just some flaky browser extension") is so valuable. In my current job, I have tried for 5+ years to get leadership to agree to a FT QA function. No dice. "Developers should test their own code." Yeah and humans should stop polluting the ocean and using fossil fuels, how's that going?
- ncphil 3y ago
  "Developers should test their own code" is emblematic of a juvenile mindset in people who regularly fire up their "reality distortion field" to avoid the effort of educating themselves on their own operations (and that helps them deny responsibility when things go South). As W. Edwards Deming, bane of all "gut instinct" executives, once wrote, "The consumer is the most important part of the production line. Quality should be aimed at the needs of the consumer, present and future." The lack of a dedicated quality team shows a lack of respect for your customers. You know, the people you need to buy your products or services (unless you're intent on living off VC loans until you have to pull the ripcord on your golden parachute).
notnmeyer 3y ago
i never felt that devops and qa were at odds the way that this article suggests they are. in my experience nobody wants to or knows how to do run QA correctly so the org shoots themselves in the foot and does one of two things: 1. hire a contractor who just has no idea about anything. 2. hire someone and place them outside the engineering org (on the support team as a "support engineer" seems pretty popular) where they have little to no interaction with either engineering _or_ customers and expect them to work miracles.
l72 3y ago
At our small tech company, QA is elevated to a whole different level. The QA lead(s) are involved in all product planning meetings and develop the requirements with the product team. Our QA lead has a phd and two have masters degrees! They know how the application is supposed to work better than most of the developers and play a big role throughout development. In my opinion (as the person that leads the developers), this is how it should work. They aren't some separate team we chunk over stuff to at the end of the day.
natbennett 3y ago
Strongly disagree with the literal premise of this post. The idea of having a separate team with the mandate to “ensure quality” was always ridiculous. Occasionally it was helpful by accident but it was always ridiculous. “Quality” isn’t something you can bake in afterwards by testing a bunch. Getting rid of everyone with testing expertise, and treating testing expertise as inherently less valuable that implementation expertise? Sure, you could convince me that was a bad idea.
- spinningD20 3y ago
  Doing every quality activity "after the fact" I agree is the issue. That's the root of the problem you're seeing, not that there was a separate quality team.
  - natbennett 3y ago
    It’s not the “separate” part that I think is ridiculous. It’s the fact that the team is named “quality assurance.” It relies on a metaphor from manufacturing that’s entirely inappropriate for software. If you want to call it “Testing and Exploration” you’d get no argument from me. (Though I do think you’ll find that team is hard to staff.)
    - spinningD20 3y ago
      I'd call it something like "Risk analysis, identification and mitigation group"
ThalesX 3y ago
I am currently working with a startup that spends a lot of time on building tests that need to be refactored every sprint because it's early stage and the assumptions change. I am shocked at the amount of developer-hours spent on tests that need to be disabled / deleted instead of just hiring 1 - 2 super cheap manual testers that just go through the flows days in and out. For me it's a no brainer, if I were CEO / CTO, until product-market-fit is achieved and exponential growth is visible, I'd just outsource Q&A and that's that.
- spinningD20 3y ago
  When outsourced, you either A) rely on someone in your org to tell them what to test and what the workflows are, ie use them as a warm body/monkey to click on things for you - this is what most people see QA as, which is silly - or B) you rely on the outsourced QA to know your product and know what is important or what all of the edge cases are. If your product is non-trivial in size or scope, ie it is not a cookie-cutter solution, then the testing of your product will also be non-trivial if you want it to work and have a good reputation (including during those all-important live demos, poc's, etc). QA does not mean "click on things and go through the happy path and everything is fine" - not saying you are implying that, but gosh the amount of companies that think it's child's play is crazy.
  - ThalesX 3y ago
    > If your product is non-trivial in size or scope Are there many products that reach such sizes without achieving Product Market Fit (PMF)? I feel like after this step is achieved, QA becomes pivotal and involves a great combination of manual and automated procedures. So I agree with you in this regard. But going back to my initial assumption. I think starting a fresh company without PMF and spending a lot on QA until that is achieved, might not be the best approach.
amaterasu 3y ago
Ignoring the common trope that developers are bad testers (I am, but not all devs are), QA presence allows teams to move faster by reducing the developer test burden to automated regression, and developer acceptance testing only. Good QA can often assist with those tasks too, further improving team velocity. Also, moving tasks to people who specialise in them is not usually a poor decision. The best way I've found to sell QA to management (especially sales/marketing/non-technical management), is to redefine them as marketing. QA output is as much about product and brand reputation management as finding bugs. IMO, nothing alienates customers faster than bugs, and bad experiences result in poor reputation. Marketing and sales people can usually assign value to passive marketing efforts, and recognise things that are damaging to retention and future sales.
pjsg 3y ago
At the start of my career (late 70s), I worked at IBM (Hursley Park) in Product Assurance (their version of QA). We wrote code and built hardware to test the product that was our area of responsibility (it was a word processing system). We wrote test cases that our code would drive against the system under test. Any issues we would describe in general terms to the development team -- we didn't want them to figure out our testcases -- we wanted them to fix the bugs. Of course, this meant that we would find (say) three bugs in linewrapping of hyphenated words and the use of backspace to delete characters, and then the development team would fix four bugs in that area but only two of the actual bugs that we had found. This meant that you could use fancy statistics to estimate the actual number of bugs left. When I've worked for organizations without QA teams, I introduce the concept of "sniff tests". This is a short (typically 1 hour) test session where anybody in the company / department is encouraged to come and bash on the new feature. The feature is supposed to be complete, but it always turns out that the edge cases just don't work. I've been in these test session where we have generated 100 bug tickets in an hour (many are duplicates). I like putting "" into every field and pressing submit. I like trying to just use the keyboard to navigate the UI. I run my system with larger fonts by default. I sometime run my browser at 110% zoom. It used to be surprising how often these simple tests would lead to problems. I'm not surprised any more!
- wrs 3y ago
  At Microsoft back in the day, we called those “bug bashes”, and my startup inherited the idea. We encouraged the whole company to take an afternoon off to participate, and gave out awards for highest impact bug, most interesting bug, etc.
  - hornban 3y ago
    This is a bit of an aside, but I have a question that I'd like to ask the wider community here. How can you do a proper bug-bash when also dealing with Scrum metrics that result in a race for new features without any regard for quality? I've tried to do this with my teams several times, but ultimately we're always coming down to the end of the sprint with too much to do to implement features, and so anybody that "takes time off" to do bug bashing looks bad because ultimately they complete fewer story points than others that don't do it? Is the secret that it only works if the entire company does it, like you suggest? And yes, I completely realize that Scrum is terrible. I'm just trying to work within a system.
pavel_lishin 3y ago
Speaking of QA, using AI to generate comic-book-style illustrations is great until one of your heroes has 6 and 8 digits per hand. At least with the comic style, you could plausibly say that that's canon to her character.
w10-1 3y ago
For QA to be respected and protected, it has to identify what it's responsible for. Luckily, that's easy: the "fault model", all the ways things can break. That tends to be a lot more complex than the operating model, the domain model, or the business model. Once all the potential issues and associated costs for all the fault models are enumerated, then QA can happily offer to any other organization the responsibility for each one, and see who steps up to take it on. In many cases, it can be done more cheaply in design, engineering, or automation; it's usually easier to prevent a problem than capture, triage, debug, fix, and re-deploy. Organizations commonly make the mistake of being oblivious to the fault models and failing to allocate responsibility. That's possible because most failures are rare, and the link from consequences back to cause is often unclear. The responsibility allocation devolves to blame, and blame to "who touched this last"? But catastrophic feedback is a terrible way to learn, and chronic irritants are among the best ways to lose customers and staff.
- bumby 3y ago
  I agree with you, but have the hunch that many PMs don't. >it's usually easier to prevent a problem than capture, triage, debug, fix, and re-deploy. It really depends on the risk of the fault. To a PM under schedule pressure, the higher risk may be to break schedule in order to redesign to mitigate the fault. As you said, many failures are low probability, so PMs are used to rolling the dice and getting away with it. Often they've moved on before those failures rear their ugly heads. An organization really needs the processes that establish guardrails against these biases. Establishing requirements to use the tools to define the fault model can go a long way, although I've seen people get away with turning a blind eye to those requirements as well. You also need to mate it with strong accountability.
nlavezzo 3y ago
When we built FoundationDB, we had a maniacal focus on quality and testing. So much so that we built it in a language we invented, called Flow, that allowed us to deterministically simulate arbitrary sized FDB clusters and subject them to crazy conditions, then flag each system property violation and be able to perfectly reproduce the test run that triggered the violation. We got to a point where the default was that all of our 10,000's of test runs each night would flash green if no new code was introduced. Tests that flashed red were almost always due to recent code additions, and therefore easily identified and fixed. It let our team develop knowing that any bugs they introduced would be quickly caught, and this translated to being able to confidently take on crazy new projects - like re-writing our transaction processing system post-launch and getting a 10x speed increase out of it. In the end our focus on quality led to velocity - they weren't mutually exclusive at all. We don't think this is an isolated phenomenon, which led us to our newest project - but that's a story for another time.
- JonChesterfield 3y ago
  Is this the language in question? https://github.com/apple/foundationdb/blob/main/flow/README.md https://github.com/apple/foundationdb/blob/main/flow/README....
  - nlavezzo 3y ago
    Yep!
rychco 3y ago
Does Medium automatically insert these AI generated images into every article now, or is that just the popular thing to do?
KaiserPro 3y ago
So for me, the QA team is the best source of product information in the entire engineering team. If they also do customer triage, then probably in the entire company. They should know the product inside out, moreover, they know all the annoying bits that are unsexy and not actively developed. yes, they find bugs and junk, but, they know how your product should be used, and the quickest/easiest way to use it. Which are often two different paths. Bring your QA in the product cycle, ask them what the stuff that pisses them off the most. They also should be the masters of writing clear and precise instructions, something devs and product owners could learn from.
guhcampos 3y ago
“ To these folks, it feels like giving a damn is a huge career liability in your organization. Because it is.” And it’s easy to see why. Software Quality, Cose Maintainability, Good Design. These things only matter if you are planning to work on that company for a long time. If you’re planning to stay a couple years then hop to the next company, the most optimal path is to rise fast by doing high visibility work, then find use your meteoric rise as a resume material to get a higher paying job. Rinse and repeat. If that project is going to break or become unmaintainable in a couple years, who cares? You’re not going to be there. Recognize the pattern? Startups work the same. It’s the “growth mindset” imprinted everywhere. If this product becomes unmaintainable in 5 years, who cares? I will have exited and cashed in. I don’t judge people who do that exactly because it’s the practice the companies themselves use. I don’t like it, I actually hate it, but I understand people are just playing by the rules. The fun part is watching managers and executives complaining about employee turnover, lack of company engagement, quiet quitting, like this isn’t them tasting their own poison.
- BoxFour 3y ago
  > Startups work the same. If this product becomes unmaintainable in 5 years, who cares? This is a reasonable stance for a startup to take. The majority of startups likely won't last five years as they tend to fail. Being alive in five years with technical debt is a good problem for most startups to have, because that means they managed to make it five years.
  - guhcampos 3y ago
    Exactly. That’s why I say I don’t like it, but understand it. I enjoy the fast paced and highly creative environments of startups more than the politics and bureaucracy of corporations, but the short term vision bothers me a lot. The result is I choose to work for midsized companies, or established startups. I kind of specialized in working on the growing pains of companies in their first hundred of engineers. There’s a lot to say about startup culture and the growth mindset, but I don’t consider it necessarily evil. It exists, lots of the products we use and love would be impossible to build without it. It can be extremely harmful, though. It burns out people, it leads to excessive risk taking, it favors aggressive, invasive marketing, it rewards reckless management - yet it works. It isn’t good or evil, like mostly everything in the World. It’s just… there.
teunispeters 3y ago
I really like working with good QA teams... ... maybe because I've never worked with a bad team. As a developer, familiarizing myself with QA processes has never been a mistake, either.
hintymad 3y ago
Maybe it's hard to find enough technically strong engineers with a passion on testing. Microsoft used to advocate how technically challenging to be a SDET - software engineer in test. IBM used to tell its test teams that testing engineers know the big picture, understand the products, and have to do awesome technical work. Unfortunately in reality, more engineers are more willing to directly develop products than becoming an expert in test.
RugnirViking 3y ago
It makes a huge difference. It kinda feels bad to have your work dissected, but it feels so great to have institutional permission to slow down and make your work good
deleted 3y ago
[deleted]
ChrisMarshallNY 3y ago
Testing has always been a huge deal with me. I tend to work alone, so I usually have to test my own stuff[0]. I’m pretty brutal. I learned from the best. I really feel that the level of testing I went through, for most of my career, would have a lot of modern devs, curled up in a fetal position, under their desks, whimpering. We are currently in a “phase 2” test, of the project we’ve worked on, for the last year or so. It has shown us some issues (nothing major, though). Phase 1 testing showed us some nasties. I had to force the team to do all this testing. They would have happily released before phase 1. I don’t think it would have ended well. [0] https://littlegreenviper.com/miscellany/testing-harness-vs-unit/ https://littlegreenviper.com/miscellany/testing-harness-vs-u...
notpachet 3y ago
I like this post, and agree with almost all of the written content. But man, those AI generated images are cancer.
dclowd9901 3y ago
A couple things I miss from not having dedicated QA: 1) someone who deeply understands how the product should work 2) someone who’s good at writing performant and maintainable tests
sharts 3y ago
The problem with QA is nobody outside of QA understands QA. They think QC is QA so they automate their QC "testing" and assume that is QA. It's not. Moreover they never listen to QA nor respect them as people. QA are usually the best people to know what's what when the rubber meets the road. They know where the bodies are buried. They often understand and have a pulse on customers and usage better than product managers. They know the ins-and-outs of how various features interact and how a product as a whole works better than silo'd developers. Historically they were effectively the only "product owners" that could certify a release before it went out. They would coordinate with the right people to ensure all technical and non-technical deliverables and dependencies were met before releases. T hey would be the best approximate and power users. They often maintained test infrastructures on which deployments could be tested. In fact, they were the origins of automation or DevOps as a "thing" because they are the ones who saw all the friction points daily giving rise to CI/CD. Often times nobody listened because they were concerned with features. QA has always been about investing resources within an organization to improve it. In effect, building things for within it to improve it. Now that we have gotten the message to some degree of optimizing some manual pain points -- we kick to the curb those who got us there without any regard to the value they provided -- and instead decide to push to prod and test on production to piss off customers even more. If you ask yourself -- would you feel comfortable driving a car that was built and tested with automation alone? What about a space shuttle or an airplane? Or about medications?Or would you prefer that a human test drivers or test pilots put those products through their paces before signing off on them? -- it might drive it home better. But then again software industry has devolved since the tech bro + VC culture pretty much ate it as they chased those sweet $$$
mynameisnoone 3y ago
It's not so much a Shakespearean "to have a QA team or not ...", but responsibility and accountability. SWEs must produce unit testing because they know the code best. Dumping this responsibility onto QA is slow, evil, and wrong like quality control only vs. QA+QC. QA teams must have the authority to ensure complicated code gets unit testing code coverage. QA teams should provide partial and full-up integration tools and testing with the support of SWEs. QA teams must have stop-the-assembly-line authority to ensure quality and testing requirements are met. QA teams (or tools teams that support multiple QA teams) must make testing faster and more efficient. There ought to be a suite of smoke tests such that untested code cannot be committed to the "production" branch, whatever that looks like, except in rare extreme emergencies. All production-ready commits should be squashed, signed-off by another SWE, and have a test plan. Test plans should be auto-generated, wherever possible. Tests combined with test infrastructure should be able to be added to auto-bisect/blame breakage Which tests must run and pass to accept a production proposed diff should be auto-minimized to those that touch particular area(s) and their precise-as-possible-but-still-correct dependencies. Other areas that must not be neglected and shoveled onto SWEs: product management, UAT, UX, and operations.
- xtracto 3y ago
  I am a CTO and have always tried to have the QA team to be part of the Product Office (CPO) . It tends to keep us honest and ultimately it aligns the decision of churning out new features and building crap in the same team/person. Now, don't get me wrong, my teams always do unit and integration testing, along with automation of those two. Devs are responsible for the quality of their work. But ultimately it os the product team, with input from their QA team the ones deciding if a new feature is ready for release as it is, or needs more polish
physicsguy 3y ago
I personally think that having worked in both environments, the biggest issue is companies missing that triage function for bugs, etc. from people who can turn a customer reported bug into a step by step guide for reproducing things. When I worked in the simulation space we used to get models sent in by customers where a convergence problem or crash would occur in 12 hours of running on a 128 core machine. Those were impossible as a developer to work with in debug mode which made the runtime even longer, so they needed someone to identify the cause of the problem and distill it down to a much smaller model where the bug could be replicated. The QA team in that were really application engineers and were subject matter experts, and they were absolutely invaluable.
bigEnotation 3y ago
>>> slowest part of software delivery is testing In my experience the slowest part has been marking a feature as done. I loved working at places with QA. I could assign tickets to QA once the PR was up. Now I gotta build in that I’ll be bumping PRs for review for approximately 30-50% of the time I’m working on a feature.
gluteart 3y ago
I joined my current company just months before they started to cut down small QA team we had then. QA Automation was supposed to be the answer. 1.5 years forward - product quality dropped, automation for the client side application barely exists and even those parts that is covered by it prone to bugs. I tried to draw attention to the fact that at least some manual QA is needed, but even after obvious fail (some people lost their job) managers are adamant. Automation, 'special bug-hunting projects', 'we should concentrate on code quality' lectures, all-hands testing - anything, instead of very obvious solution to get QA team back. Development time is up, regression is often, communication became harder. The only QA who still works in the company (now in a different role) became invaluable, because he is one of the very few people who deeply understands the product as a whole and knows all the services we work with. I can't think of another example of so very obvious mistake and solution to it, that's being ignored so relentless.

433 points by nlavezzo 3y ago

deleted 3y ago

[deleted]

In making the case for building up a QA org at my current startup, I repeat the mantra that QA is both a skillset and a mindset. Automated tests can tell us a lot, but skilled QA testers are amazing at edge cases to break things and providing human feedback about what looks and feels good gor users.

robotnikman 3y ago

Microsoft is probably the one case where this sticks out the most, at least for me anyways. Noticeably more bugs in updates since they dropped their QA team, in Windows as well as cloud products.

EvanAnderson 3y ago
The revenue keeps rolling in so, clearly, they made the right business decision... >sigh< I use a lot of MSFT software and services in the "day job". I wish there was some kind of consequence to them for their declining quality.
- danesparza 3y ago
  Almost like you should be able to file a 'bug report' or something. Maybe they should build a team to make sure their code quality is up to snuff...
- shiroiuma 3y ago
  >The revenue keeps rolling in so, clearly, they made the right business decision... >sigh< This is exactly the point I was going to make. Their stock price is doing great! So obviously they've done the right thing for their position in the market: the market has rewarded them for not wasting money on QA and just letting users suffer with the bugs. >I wish there was some kind of consequence to them for their declining quality. If people keep insisting on throwing money at them no matter how bad their software is, then there's no reason for them to improve their quality.

alanjay 3y ago

But Microsoft fired their entire qa department, and their software is rock solid. Right! Right?

deleted 3y ago
[deleted]

mschuster91 3y ago

Another part is that there is barely any training for QA people. Even your average CS course will only graze the top of the topic, most usually some prof droning on about Java unit tests on some really old version of Java and a testing framework just as old. There are no "Software Quality Assurance" academic degrees, there's barely any research into testing methodologies, there's barely any commercial engagement in the space aside from test run environments (aka, selling shovels to gold diggers), and let's face truth, also in tooling. And everything but software QA is an even worse state, with "training" usually consisting of a few weeks of "learning on the job". Basic stuff like "how does one even write a test plan", "how does one keep track of bugs", "how to determine what needs to be tested in what way (unit, integration, e2e)" is at best cargo-culted in the organization, at worst everyone is left to reinvent the wheel themselves, and you end up with 13 different "testing" jobs in a manually clicked-together Jenkins server, for one project. > Defect Investigation: Reproduction, or “repro”, is a critical part of managing bugs. In order to expedite fixes, somebody has to do the legwork to translate “I tried to buy a movie ticket and it didn’t work” into “character encoding issues broke the purchase flow for a customer with a non-English character in their name”. And this would normally not be the job of a QA person, that's 1st level support's job, but outsourcing to Indian body shops or outright AI chatbots is cheaper than hiring competent support staff. That also ties in to another aspect I found lacking in the article: users are expected to be your testers for free aka you sell bananaware. No matter if it's AAA games, computer OSes, phones, even cars...

wmichelin 3y ago
To play devil's advocate here, I did not _really_ get training for my software engineering role. I got a little bit from 1-2 college courses, but the vast majority of my role I had to pick up on the job or on my own. I can tell you, I definitely didn't get training for QA tasks, but here I am doing them anyways. It's just work that needs to be done.
- mschuster91 3y ago
  > To play devil's advocate here, I did not _really_ get training for my software engineering role Yeah and that is my point. It would be way better for the entire field of QA if there were at least a commonly agreed base framework of concepts and ways to do and especially to name things, if alone because the lack of standardization wrecks one's ability to even get testers and makes onboarding them to your "in house standards" a very costly endeavour.

wmichelin 3y ago

This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering. I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests". With this model, I have found that the engineering team ends up being the QA team in practice, and then the actual QA team often only finds bugs that aren't really bugs, just creating noise and taking away more value than they provide. I would love to learn about QA team models that work. Manual tests are great, but they only go so far in my experience. I'm not trying to knock on QA folks, I'm just sharing my experience.

convolvatron 3y ago
in the classic model, most QA orgs were a useless appendage. partially by construction, but largely because QA gets squeezed out when dev is late (when does that happen?). they aren't folded in early, so they twiddle their thumbs doing 'test infrastructure' and 'test plans', until they finally get a code drop and a 48 hr schedule to sign off, which they are under extreme pressure to do. but every once and a while you ran across a QA organization that actually had a deep understanding of the problem domain, and actually helped drive development. right there alongside dev the entire way. not only did they improve quality, but they actually saved everyone time.
- lambic 3y ago
  Not sure why this was downvoted, that second paragraph is right on the money.
- cratermoon 3y ago
  Saying "useless appendage" sounds to me like it's the QA team that's the problem, when what you're really saying is that it's the organization and process that pushed QA teams into irrelevance. I agree with your assessment overall, and those issues were one of the driving forces behind companies dispensing with QA and putting it all on the developers.
- 3y ago

cloths 3y ago

I can't agree more. > Focus: There is real value in having people at your company whose focus is on the quality of your end product. Quality might be “everybody’s job”…but it should also be “somebody’s job”. Yes indeed, naturally every person have just one focus, having dedicated person focus on QA is important. Another practice, or buzz word (or used to be buzz word:) ), Exploratory Testing, which can pretty much be conducted only by dedicated QA.

philk10 3y ago
That's pretty much my role - I don't write test cases, I'll explore the system and try to find issues that the devs have missed. Then they learn from what they missed so I have to explore more to find other types of issues.

gyudin 3y ago

Getting rid of QA teams are a slow ticking bomb frequently imho. Cause some issues are might not be even breaking functionality. You can mess up some tracking/analytics and managers will make wrong decisions based on incorrect data. But personally I feel like within few years everything might change a lot. Machines 100% will be be better at coding, maintenance and testing things.

harshalizee 3y ago

This is a symptom of a larger issue in tech where C/E suites are trying really hard to turn engineers into some sort of fungible cogs in the system that can be swapped in and out and in different parts of the system and still have everything work perfectly in order.

righthand 3y ago
Yep I see this a lot where people want swappable engineers and no one is able to understand if you have an engineer working in the Frontend most of the time they will not be acquainted with backend work. Nor is there a need or logical way to keep everyone working across the stack to keep them in this swappable state. Each time you change a persons job they need retraining and orientation. Code is code but a drop down menu is not a database insert.

righthand 3y ago

QA Engineers are some of the best debuggers too. They have their hands in the pipeline, src and test directories, and often work with all aspects of developing and deploying the application. When I was a QA lead I often ran into software engineers that couldn’t be bothered to read a pipeline error message (and would complain daily in Slack) and when it came to optimizing the pipeline they would ignore base problems and pretend the issues that stemmed from the base problems were magical and not understood. Wasting days guessing at a solution. The disrespect a QA engineer sees is not exaggerated in this article. Since most companies with QA orgs do not have a rigorous interviewing process like the Engineering orgs, the QA engineers are seen as lesser. The only SWE that have respect for them that I’ve met are the people who worked in QA themselves. The disrespect is so rampant that I myself have switched back to the Engineering org (I tried using seniority as a principal engineer and even shifted as a manager to make changes, but this failed because Engineering could not see past their own hubris and leadership peoples will not help you). My previous company before I was laid off hired a new CTO who claimed we could just automate away QA needs but had no examples of what she was talking about. This is the level of respect poured down from the top about building good software.

deleted 3y ago
[deleted]

ptmcc 3y ago

I stepping-stoned through QA on my way into development, now a decade something ago, and this part stands out as especially true in my experience: > This created a self-reinforcing spiral, in which anyone “good enough at coding” or fed up with being treated poorly would leave QA. Similarly, others would assume anyone in QA wasn’t “good enough” to exit the discipline. No one recommends the field to new grads. Eventually, the whole thing seemed like it wasn’t worth it any more. Our divorce with QA was a cold one — companies just said “we’re no longer going to have that function, figure it out.” I've worked with a handful of talented career software QA people in the past. The sanity they can bring to a complex system is amazing, but it seems like a shrinking niche. The writing was on the wall for me that I needed and wanted to get into fulltime dev as QA got increasingly squeezed, mistreated, and misunderstood. At so many companies QA roles went into a death spiral and haven't recovered. Now, as the author points out, a lot of younger engineers have never worked with a QA team in their career. Or maybe have worked with a crappy QA team because its been devalued so much. So many people have never seen good QA that no one advocates for it.

reactordev 3y ago
>anyone in QA wasn’t “good enough” This is why. Engineers have some of the most inflated egos that they set an extremely high bar for being “part of the club”. Sometimes that’s corporate policy (hire better than you) and sometimes it’s just toxicity (I am better than you). Without realizing that the most valuable skills they could learn are soft skills. I’m open to finding anyone willing to code. Whether it’s from QA, sales, Gerry’s nephew, recent CS grad, Designer turned coder, or that business analyst that taught themselves Python/Pandas. A good QA team is sorely missed. A bad QA team turns the whole notion of QA teams sour. Just the same for development teams :D I think devs are first line of defense. Unit tests etc. QA is second line (should we release?), feature testing, regression, UX continuity, etc. There’s value in it if you can afford it.

annoyingnoob 3y ago

I remember a time before agile and devops. Seems like QA has always been looked down on, and always considered a bottleneck.

oaththrowaway 3y ago

I had a boss at Yahoo who gave our QA to another team because "Facebook doesn't use QA, we shouldn't either". I can't remember if it was Facebook or MS, but he was willing to buy all of us a book talking about how amazing it was. Long story short, it wasn't. It was like taking away a crutch. Of course we could have been more diligent about testing before having QA validate it, but it slowed development down so much trying to learn all the things we never thought to test that QA did automatically.

robocat 3y ago
An article about Facebook's reason for no QA with some of the mitigations: https://blog.southparkcommons.com/move-fast-or-die/ https://blog.southparkcommons.com/move-fast-or-die/ A bit recent to have affected Yahoo - but it sells a good story. We would celebrate the first time someone broke something. Let anyone touch any part of the codebase and get in there to fix a bug or build a feature. Yes, this can cause bugs. Yes, that is an acceptable tradeoff. We had a shared mythology about the times that the site went down and we all rallied together to quickly bring it back up. Sounds like hell: running as close to the edge of the cliff as you can. Presumably totally ignoring thousands of papercuts of slightly broken functionality. Optimising to produce an infinite number of shallow bugs.
- quadrifoliate 3y ago
  The difference is that bugs in the social network parts of Facebook (the ones where you see your friends and family's pictures and posts) are not directly making money for Facebook. The only real stuff that matters for the money is all the tracking. I bet the people responsible for Facebook Ads Manager are a lot less enthusiastic about "move fast and break things", although I'd be interested to hear an opposing viewpoint from anyone here who's worked for that group.

amlozano 3y ago

This point is brought up in the article but I think it is at the real heart of the issue. QA is almost always seen as a 'cost center' by the business and upper management. I have a hypothesis that you never ought to work in a department that is seen as a 'cost center'. The bonuses, the recognition, and the respect always goes to the money makers. The cost center is the first place to get more work with less hands, get blamed for failures, and ultimately fired when the business needs to slim up. I think the same thing applies to IT. This spiral is why QA will always be a harder career than just taking similar skills and being a developer. It self reinforces that the best people get fed up and switch out as soon as they can.

robofanatic 3y ago
> QA is almost always seen as a 'cost center' by the business and upper management Well everything involved in making a product is seen as a cost, that includes the entire development team - QA, Developers, Devops, PM ....
- rubidium 3y ago
  No. That’s not actually how most orgs break it down. R&D, marketing, sales is “bringing new business” so are profit centers. This means their budget grows with revenue. Manufacturing, QA, IT and service are cost centers so get squeezed year-over-year even if revenue is flat.
  - itsdrewmiller 3y ago
    What software companies are not classifying QA as R&D?
  - jdlshore 3y ago
    It depends on the org. In my company, which is a SaaS-like company, all of product and engineering is a cost center, despite creating the product the company sells. It’s just the way they do their accounting.
serial_dev 3y ago

hasoleju 3y ago

I completely agree with the sentiment of this article: It is a big problem that being a "software tester" is not at all as prestigious as being a software engineer. Having someone who really understands how the users interact with the software and systematically covers all behavior in test cases is very valuable. I experienced both worlds: I worked in an organization where 4 QA engineers tested each release that was built by 6 software engineers. Now I'm in a situation where 0 QA engineers test the work of 8 software engineers. In the second case the software engineers actually do all the testing, but not that systematically because it's not their job to optimize the testing process. Having someone with the capabilities of a software engineer who's daily work is uncovering defects and verifying functionality is important. Paying someone who owns the testing process is more than justified commercially. The problem is: You don't find those people. For various reasons. Therefor you are stuck with making the software engineers do the testing. But there is hope. There is a new standard released for my industry that requires organizations to have a QA department that is independent of the software engineering department. If they don't have that, they are not allowed to role out there software product complaint to the standard. Maybe this will help to reintroduce the QA Engineer as an important and prestigious role.

zabzonk 3y ago
what industry is that?
- hasoleju 3y ago
  Mechanical engineering and shopfloor software for factory automation in Europe. There is a new IT security standard released that also includes requirements for the software.

godelski 3y ago

The main problem with QA teams is the same problem with IT teams or even management. If they are doing their jobs well they appear to be doing nothing. This often creates a situation where people need to "justify" their jobs. Usually this happens due to an over reliance upon metrics (see Goodhart's Law) rather than understanding what the metrics are proxying and what the actual purpose of the job is. A bad QA team is one who is overly nitpicky, looking to ensure they have something to say. A good QA team simultaneous checks for quality as well as trains employees to produce higher quality. I do feel like there is a lack of training going on in the workforce. We had the "95%-ile isn't that good"[0] post on the front page not long ago and literally it is saying "It's easy to get to the top 5% of performers in any field because most performers don't actively train or have active feedback." It's like the difference between being on an amateur sports team vs a professional. Shouldn't businesses be operating like the latter? Constantly training? Should make hiring be viewed differently too, as in "can we turn this person into a top performer" rather than "are they already" because the latter isn't as meaningful as it appears when your environment is vastly different than the one where success was demonstrated. [0] https://news.ycombinator.com/item?id=38560345 https://news.ycombinator.com/item?id=38560345

philk10 3y ago
When I work with new devs I can often trip them up with basic tests of double-clicking, leading spaces, using the back button on Android. They then learn these and from then on these issues dont appear ( well, OK, it might take a couple of times of a ticket being rejected because of these but they do quickly learn all my tricks ) I don't get measured on bugs found so there's no pressure on me to find stupid bugs just to boost my figures.
- hotpotamus 3y ago
  I had a buddy who liked to do that kind of thing. I think his favorite trick was just to enter nothing in a form and hit enter to see what happens. It's probably his favorite because it deleted the database on some little thing I wrote at one point and we got a good laugh over it and I got a good lesson out of it.

itqwertz 3y ago

I noticed this trend a couple of years ago, they called it Shift-Left. Basically, it was a way to get a developer to do more testing during development before handing over a feature to QA. This sets up the imagined possibility of firing all QA staff and having developers write perfect code that never needs to be tested thoroughly. Looks great on paper... At a previous company, they started firing all of the manual QA devs and replacing them with offshore QA people who could do automated testing development (Cypress tests). The only problem was that those fired QA team members had significant business domain knowledge that never was transferred or written down. The result was a dumpster fire every week, with the usual offshore nightmares and panicked responses. Make no mistake about this, it's just a cost-cutting measure with an impact that is rarely felt immediately by management. I've worked with stellar manual QA people and our effort resulted in bulletproof software that handled edge cases, legacy customers, API versions, etc. Without these people, your software eventually becomes IT-IS-WHAT-IT-IS and your position is always threatened due to the lack of quality controls.

deleted 3y ago

[deleted]

evilantnie 3y ago

QA has always been about risk management. There are multiple ways to manage risk, and some of those ways can be more cost effective to a business. As software shifted towards SaaS offerings, deployments (and rollbacks) became quicker, customer feedback loops also got lightning fast. Team's can manage the risk of a bug more efficiently by optimizing for mean-time-to-recovery. This muscle is not one that QA teams are particularly optimized for, thus their effectiveness in this new model was reduced. I've found that holding on to QA function in this environment can severely dilute the ownership of quality as a requirement from engineers. QA is still extremely valuable in any software that has long deployment lead times. Mobile apps, On-Prem solutions, anything that cannot be deployed or rolled back within minutes can benefit from a dedicated QA team that can manage the risk appropriately.

spinningD20 3y ago
There are so so many instances where "rolling back" is just not a feasible solution. Working for a SaaS company with mobile/web/api and huge db's, migrations, payroll uses in the product, rolling back is and should always be a LAST RESORT. In 99% of the cases, something significant enough to want to roll back usually results in a "hot patch" workflow instead because rolling back or etc has its own risk. > QA has always been about risk management. 100%. QA should be related to identifying risk, likelihood of failure, impact of failure to user, client and company. The earlier this is done in the varying processes, the better. ("shift left" but I've seen a ton of differences with how people describe this, but generally QA should start getting involved in the "design phase") Another example from my own first-hand experience: A company I worked for made a product that plugged into machines that were manufacturing parts, and based on your parameters it would tell you whether or not the part was "good" or "bad". When interviewing the leadership of the company, as well as the majority of the engineering group, "what is the biggest risk with this product" they all said "if the product locks up!". Upon further discussion, I pulled out a much larger, insidious risk; "what if our product tells the client that the part is 'good' when it is not?" In this example, the part could be involved in a medical device that keeps someone alive. You're not going to be able to roll that back.

blastbking 3y ago

Agree with the sentiment of this article but the disturbing ai generated images every paragraph were definitely not necessary - do people actually need to see these?

jawns 3y ago
It did strike me as ironic that the article is about ruthlessly automating to avoid paying QA engineers, and it uses AI to avoid paying illustrators.
- willsmith72 3y ago
  not saying this is you, but i get so tired of feedback about ai-generated images along the lines of "you're taking money away from local artists" it's not one or the other. in my experience it's a decision of "no images" vs "ai images". in this case, probably "no images" would've been better for the reading experience. but there was never any illustrator getting paid
Night_Thastus 3y ago
I'd rather have stock or existing images, or none at all. Every time I looked at one of those AI images my brain just kept seeing all the little weird parts that didn't make sense. Like a brain itch.

somewhereoutth 3y ago

Systems used by humans have to be tested by humans. Those testers can either be your customers, or your QA team - as your dev and sales teams will be busy doing other things.

amtamt 3y ago

> The most conscientious employees in your organization are the most bitter. They see the quality issues, they often address them, and they get no recognition for doing so. When they speak up about quality concerns, they get treated like mouthbreathers who want to slow down. They watch the “move fast and break things” crowd get rewarded time after time, while they run around angrily cleaning up their messes. To these folks, it feels like giving a damn is a huge career liability in your organization. Because it is. This is the bitter truth, no one wants to acknowledge. DBAs and Infra, are in the same boat as QAs. Pendulam will swing back in not so long time frame i hope.

jabroni_salad 3y ago
My idea of the pendulum swinging back is 'you build it you run it', personally. Don't like the oncall pager? don't make it ring.
- hinkley 3y ago
  The hard part for most people is learning to tell the devil in a necktie to fuck off every time they try to sweet talk you into volunteering for that kind of pain. Also with no recognition or compensation. All it takes is one person on my team to defect and support an untenable amount of tech debt, and everyone on my team has to pay for it.
esafak 3y ago
No, just jump ship and let the damn company fail. Fail faster, haha! This is how we have nice things; when bad companies are not propped up.
sporedro 3y ago
While QA and testing is important, I’m not sure you can convince the people only concerned about profits… I think the “release it fast and patch it later” concept here to stay due to the internet being so accessible. Why bother spending tons of money and time when the users will just report the bugs and you can release updates over the internet they can download. Ever since physical copies of video games and software were replaced mainly by downloads, it seems like patching is cheaper. Of course this leads to horrendous security issues, bad user experience, etc. but who cares as long as the guy on top is maximizing profits.

karmakaze 3y ago

Having a QA team is like having an Ops team with stuff being 'thrown over the wall' to the downstream. There's two kinds of tests. Regression testing, that should be automated and written and maintained by devs. New feature or change testing should be done by those that defined them, namely Product people. In the best case it's an iterative and collaborative process, where things can be evaluated in dev/test environments, staging environments, or production for beta flag enabled test users.

Zelphyr 3y ago

I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path. One key attribute to both companies is that it was dictated from on high that the QA team had final say whether the release went to production or not. These days companies think having the developers write automated tests and spend an inordinate amount of time worrying over code coverage is better. I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work. I'm not saying automated testing is bad. I'm saying, just as the author does, that doing away with human QA testers is.

supportengineer 3y ago
I've seen QA/QE greatness and it was similar to how you describe. A different chain of command for deciding if releases are certified for production. Different incentive structures as well. Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.
- eitally 3y ago
  I had a team like that once. It was glorious. And ultimately, I'm convinced it led to overall faster development cycles because the baseline code quality & documentation was so much better than it would have been without such a great QA manager. The QA team, of course, was also technical -- mostly with SWE backgrounds -- and they were primarily colo'd in the same office as the dev team. I still remember the epiphany everyone had one planning cycle when it was mutually understood that by generally agreeing to use TDD, the QA team could participate actively in the real engineering planning and product development process. ... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".

RedShift1 3y ago

Now just to convince msft of this fact because their shit's been breaking left and right and all over the place.

temuze 3y ago

I strongly disagree. I worked at a company with a world-class QA team. They were amazing and I can't say enough nice things about them. They were comprehensive and professional and amazing human beings. They had great attention to detail and they catalogued a huge spreadsheet of manual things to test. Engineers loved working with them. However -- the end result was that engineers got lazy. They were throwing code over to QA while barely testing code themselves. They were entirely reliant on manual QA, so every release bounced back and forth several times before release. Sometimes, we had feature branches being tested for months, creating HUGE merge conflicts. Of course, management noticed this was inefficient, so they formed another team dedicated to automated QA. But their coverage was always tiny, and they didn't have resources to cover every release, so everyone wanted to continue using manual QA for CYA purposes. When I started my own company, I hired some of my old engineering coworkers. I decided to not hire QA at all, which was controversial because we _loved_ our old QA team. However, the end result was that we were much faster. 1. It forced us to invest heavily on automation (parallelizing the bejesus out of everything, so it runs in <15min), making us much faster 2. Engineers had a _lot_ of motivation to test things well themselves because there was no CYA culture. They couldn't throw things over a wall and wash their hands of any accountability. We also didn't have a lack of end-to-end tests, as the author alludes to. Almost all of our tests were functional / integration tests, that run on top of a docker-compose set up that simulated production pretty well. After all, are unit tests where you mock every data source helpful at all? We invested a lot of time in making realistic fixtures. Sure, we released some small bugs. But we never had huge, show stopping bugs because engineers acted as owners, carefully testing the worst-case scenarios themselves. The only downside was that we were slower to catch subtle, not-caught-by-Sentry bugs, so things like UX transition weirdness. But that was mostly okay. Now, there is still a use case for manual QA -- it's a question of risk tolerance. However, most applications don't fit that category.

mixmastamyk 3y ago
False dichotomy. Poor dev practice is not fixed by elimination of QA, but rather fixed by improving dev practice. The “five why’s” can help.
- 3y ago

Ensorceled 3y ago

I hired a great QA Developer a few months ago. They are building out integration, performance and end to end tests; finding bugs, speeding up releases and generally making things better. I get asked, every week, if they are contributing and what are they contributing. It's exhausting, so I can't imagine what it feels like to actually BE in QA.

warkdarrior 3y ago
# of bugs found per week should be sufficient metric of productivity.

bgribble 3y ago

I was lucky enough to work in a small eng team with 1 full-time dedicated QA person. One of the very few coworkers from my long career that I have really tried hard to poach away from whatever they were doing after our shared workplace went bust. Yes, part of the job was to write and run manual test suites, and to sign off as the DRI that a certain version had had all the automated and manual tests pass before release. But their main value was in the completely vague mandate "get in there and try to break it." Having someone who knows the system and can really dig into the weak spots to find problems that devs will try to handwave away ("just one report of the problem? probably just some flaky browser extension") is so valuable. In my current job, I have tried for 5+ years to get leadership to agree to a FT QA function. No dice. "Developers should test their own code." Yeah and humans should stop polluting the ocean and using fossil fuels, how's that going?

ncphil 3y ago
"Developers should test their own code" is emblematic of a juvenile mindset in people who regularly fire up their "reality distortion field" to avoid the effort of educating themselves on their own operations (and that helps them deny responsibility when things go South). As W. Edwards Deming, bane of all "gut instinct" executives, once wrote, "The consumer is the most important part of the production line. Quality should be aimed at the needs of the consumer, present and future." The lack of a dedicated quality team shows a lack of respect for your customers. You know, the people you need to buy your products or services (unless you're intent on living off VC loans until you have to pull the ripcord on your golden parachute).

notnmeyer 3y ago

i never felt that devops and qa were at odds the way that this article suggests they are. in my experience nobody wants to or knows how to do run QA correctly so the org shoots themselves in the foot and does one of two things: 1. hire a contractor who just has no idea about anything. 2. hire someone and place them outside the engineering org (on the support team as a "support engineer" seems pretty popular) where they have little to no interaction with either engineering _or_ customers and expect them to work miracles.

l72 3y ago

At our small tech company, QA is elevated to a whole different level. The QA lead(s) are involved in all product planning meetings and develop the requirements with the product team. Our QA lead has a phd and two have masters degrees! They know how the application is supposed to work better than most of the developers and play a big role throughout development. In my opinion (as the person that leads the developers), this is how it should work. They aren't some separate team we chunk over stuff to at the end of the day.

natbennett 3y ago

Strongly disagree with the literal premise of this post. The idea of having a separate team with the mandate to “ensure quality” was always ridiculous. Occasionally it was helpful by accident but it was always ridiculous. “Quality” isn’t something you can bake in afterwards by testing a bunch. Getting rid of everyone with testing expertise, and treating testing expertise as inherently less valuable that implementation expertise? Sure, you could convince me that was a bad idea.

spinningD20 3y ago
Doing every quality activity "after the fact" I agree is the issue. That's the root of the problem you're seeing, not that there was a separate quality team.
- natbennett 3y ago
  It’s not the “separate” part that I think is ridiculous. It’s the fact that the team is named “quality assurance.” It relies on a metaphor from manufacturing that’s entirely inappropriate for software. If you want to call it “Testing and Exploration” you’d get no argument from me. (Though I do think you’ll find that team is hard to staff.)
  - spinningD20 3y ago
    I'd call it something like "Risk analysis, identification and mitigation group"

ThalesX 3y ago

I am currently working with a startup that spends a lot of time on building tests that need to be refactored every sprint because it's early stage and the assumptions change. I am shocked at the amount of developer-hours spent on tests that need to be disabled / deleted instead of just hiring 1 - 2 super cheap manual testers that just go through the flows days in and out. For me it's a no brainer, if I were CEO / CTO, until product-market-fit is achieved and exponential growth is visible, I'd just outsource Q&A and that's that.

spinningD20 3y ago
When outsourced, you either A) rely on someone in your org to tell them what to test and what the workflows are, ie use them as a warm body/monkey to click on things for you - this is what most people see QA as, which is silly - or B) you rely on the outsourced QA to know your product and know what is important or what all of the edge cases are. If your product is non-trivial in size or scope, ie it is not a cookie-cutter solution, then the testing of your product will also be non-trivial if you want it to work and have a good reputation (including during those all-important live demos, poc's, etc). QA does not mean "click on things and go through the happy path and everything is fine" - not saying you are implying that, but gosh the amount of companies that think it's child's play is crazy.
- ThalesX 3y ago
  > If your product is non-trivial in size or scope Are there many products that reach such sizes without achieving Product Market Fit (PMF)? I feel like after this step is achieved, QA becomes pivotal and involves a great combination of manual and automated procedures. So I agree with you in this regard. But going back to my initial assumption. I think starting a fresh company without PMF and spending a lot on QA until that is achieved, might not be the best approach.

amaterasu 3y ago

Ignoring the common trope that developers are bad testers (I am, but not all devs are), QA presence allows teams to move faster by reducing the developer test burden to automated regression, and developer acceptance testing only. Good QA can often assist with those tasks too, further improving team velocity. Also, moving tasks to people who specialise in them is not usually a poor decision. The best way I've found to sell QA to management (especially sales/marketing/non-technical management), is to redefine them as marketing. QA output is as much about product and brand reputation management as finding bugs. IMO, nothing alienates customers faster than bugs, and bad experiences result in poor reputation. Marketing and sales people can usually assign value to passive marketing efforts, and recognise things that are damaging to retention and future sales.

pjsg 3y ago

At the start of my career (late 70s), I worked at IBM (Hursley Park) in Product Assurance (their version of QA). We wrote code and built hardware to test the product that was our area of responsibility (it was a word processing system). We wrote test cases that our code would drive against the system under test. Any issues we would describe in general terms to the development team -- we didn't want them to figure out our testcases -- we wanted them to fix the bugs. Of course, this meant that we would find (say) three bugs in linewrapping of hyphenated words and the use of backspace to delete characters, and then the development team would fix four bugs in that area but only two of the actual bugs that we had found. This meant that you could use fancy statistics to estimate the actual number of bugs left. When I've worked for organizations without QA teams, I introduce the concept of "sniff tests". This is a short (typically 1 hour) test session where anybody in the company / department is encouraged to come and bash on the new feature. The feature is supposed to be complete, but it always turns out that the edge cases just don't work. I've been in these test session where we have generated 100 bug tickets in an hour (many are duplicates). I like putting "" into every field and pressing submit. I like trying to just use the keyboard to navigate the UI. I run my system with larger fonts by default. I sometime run my browser at 110% zoom. It used to be surprising how often these simple tests would lead to problems. I'm not surprised any more!

wrs 3y ago
At Microsoft back in the day, we called those “bug bashes”, and my startup inherited the idea. We encouraged the whole company to take an afternoon off to participate, and gave out awards for highest impact bug, most interesting bug, etc.
- hornban 3y ago
  This is a bit of an aside, but I have a question that I'd like to ask the wider community here. How can you do a proper bug-bash when also dealing with Scrum metrics that result in a race for new features without any regard for quality? I've tried to do this with my teams several times, but ultimately we're always coming down to the end of the sprint with too much to do to implement features, and so anybody that "takes time off" to do bug bashing looks bad because ultimately they complete fewer story points than others that don't do it? Is the secret that it only works if the entire company does it, like you suggest? And yes, I completely realize that Scrum is terrible. I'm just trying to work within a system.

pavel_lishin 3y ago

Speaking of QA, using AI to generate comic-book-style illustrations is great until one of your heroes has 6 and 8 digits per hand. At least with the comic style, you could plausibly say that that's canon to her character.

w10-1 3y ago

For QA to be respected and protected, it has to identify what it's responsible for. Luckily, that's easy: the "fault model", all the ways things can break. That tends to be a lot more complex than the operating model, the domain model, or the business model. Once all the potential issues and associated costs for all the fault models are enumerated, then QA can happily offer to any other organization the responsibility for each one, and see who steps up to take it on. In many cases, it can be done more cheaply in design, engineering, or automation; it's usually easier to prevent a problem than capture, triage, debug, fix, and re-deploy. Organizations commonly make the mistake of being oblivious to the fault models and failing to allocate responsibility. That's possible because most failures are rare, and the link from consequences back to cause is often unclear. The responsibility allocation devolves to blame, and blame to "who touched this last"? But catastrophic feedback is a terrible way to learn, and chronic irritants are among the best ways to lose customers and staff.

bumby 3y ago
I agree with you, but have the hunch that many PMs don't. >it's usually easier to prevent a problem than capture, triage, debug, fix, and re-deploy. It really depends on the risk of the fault. To a PM under schedule pressure, the higher risk may be to break schedule in order to redesign to mitigate the fault. As you said, many failures are low probability, so PMs are used to rolling the dice and getting away with it. Often they've moved on before those failures rear their ugly heads. An organization really needs the processes that establish guardrails against these biases. Establishing requirements to use the tools to define the fault model can go a long way, although I've seen people get away with turning a blind eye to those requirements as well. You also need to mate it with strong accountability.

nlavezzo 3y ago

When we built FoundationDB, we had a maniacal focus on quality and testing. So much so that we built it in a language we invented, called Flow, that allowed us to deterministically simulate arbitrary sized FDB clusters and subject them to crazy conditions, then flag each system property violation and be able to perfectly reproduce the test run that triggered the violation. We got to a point where the default was that all of our 10,000's of test runs each night would flash green if no new code was introduced. Tests that flashed red were almost always due to recent code additions, and therefore easily identified and fixed. It let our team develop knowing that any bugs they introduced would be quickly caught, and this translated to being able to confidently take on crazy new projects - like re-writing our transaction processing system post-launch and getting a 10x speed increase out of it. In the end our focus on quality led to velocity - they weren't mutually exclusive at all. We don't think this is an isolated phenomenon, which led us to our newest project - but that's a story for another time.

JonChesterfield 3y ago
Is this the language in question? https://github.com/apple/foundationdb/blob/main/flow/README.md https://github.com/apple/foundationdb/blob/main/flow/README....
- nlavezzo 3y ago
  Yep!

rychco 3y ago

Does Medium automatically insert these AI generated images into every article now, or is that just the popular thing to do?

KaiserPro 3y ago

So for me, the QA team is the best source of product information in the entire engineering team. If they also do customer triage, then probably in the entire company. They should know the product inside out, moreover, they know all the annoying bits that are unsexy and not actively developed. yes, they find bugs and junk, but, they know how your product should be used, and the quickest/easiest way to use it. Which are often two different paths. Bring your QA in the product cycle, ask them what the stuff that pisses them off the most. They also should be the masters of writing clear and precise instructions, something devs and product owners could learn from.

guhcampos 3y ago

“ To these folks, it feels like giving a damn is a huge career liability in your organization. Because it is.” And it’s easy to see why. Software Quality, Cose Maintainability, Good Design. These things only matter if you are planning to work on that company for a long time. If you’re planning to stay a couple years then hop to the next company, the most optimal path is to rise fast by doing high visibility work, then find use your meteoric rise as a resume material to get a higher paying job. Rinse and repeat. If that project is going to break or become unmaintainable in a couple years, who cares? You’re not going to be there. Recognize the pattern? Startups work the same. It’s the “growth mindset” imprinted everywhere. If this product becomes unmaintainable in 5 years, who cares? I will have exited and cashed in. I don’t judge people who do that exactly because it’s the practice the companies themselves use. I don’t like it, I actually hate it, but I understand people are just playing by the rules. The fun part is watching managers and executives complaining about employee turnover, lack of company engagement, quiet quitting, like this isn’t them tasting their own poison.

BoxFour 3y ago
> Startups work the same. If this product becomes unmaintainable in 5 years, who cares? This is a reasonable stance for a startup to take. The majority of startups likely won't last five years as they tend to fail. Being alive in five years with technical debt is a good problem for most startups to have, because that means they managed to make it five years.
- guhcampos 3y ago
  Exactly. That’s why I say I don’t like it, but understand it. I enjoy the fast paced and highly creative environments of startups more than the politics and bureaucracy of corporations, but the short term vision bothers me a lot. The result is I choose to work for midsized companies, or established startups. I kind of specialized in working on the growing pains of companies in their first hundred of engineers. There’s a lot to say about startup culture and the growth mindset, but I don’t consider it necessarily evil. It exists, lots of the products we use and love would be impossible to build without it. It can be extremely harmful, though. It burns out people, it leads to excessive risk taking, it favors aggressive, invasive marketing, it rewards reckless management - yet it works. It isn’t good or evil, like mostly everything in the World. It’s just… there.

teunispeters 3y ago

I really like working with good QA teams... ... maybe because I've never worked with a bad team. As a developer, familiarizing myself with QA processes has never been a mistake, either.

hintymad 3y ago

Maybe it's hard to find enough technically strong engineers with a passion on testing. Microsoft used to advocate how technically challenging to be a SDET - software engineer in test. IBM used to tell its test teams that testing engineers know the big picture, understand the products, and have to do awesome technical work. Unfortunately in reality, more engineers are more willing to directly develop products than becoming an expert in test.

RugnirViking 3y ago

It makes a huge difference. It kinda feels bad to have your work dissected, but it feels so great to have institutional permission to slow down and make your work good

deleted 3y ago

[deleted]

ChrisMarshallNY 3y ago

Testing has always been a huge deal with me. I tend to work alone, so I usually have to test my own stuff[0]. I’m pretty brutal. I learned from the best. I really feel that the level of testing I went through, for most of my career, would have a lot of modern devs, curled up in a fetal position, under their desks, whimpering. We are currently in a “phase 2” test, of the project we’ve worked on, for the last year or so. It has shown us some issues (nothing major, though). Phase 1 testing showed us some nasties. I had to force the team to do all this testing. They would have happily released before phase 1. I don’t think it would have ended well. [0] https://littlegreenviper.com/miscellany/testing-harness-vs-unit/ https://littlegreenviper.com/miscellany/testing-harness-vs-u...

notpachet 3y ago

I like this post, and agree with almost all of the written content. But man, those AI generated images are cancer.

dclowd9901 3y ago

A couple things I miss from not having dedicated QA: 1) someone who deeply understands how the product should work 2) someone who’s good at writing performant and maintainable tests

sharts 3y ago

The problem with QA is nobody outside of QA understands QA. They think QC is QA so they automate their QC "testing" and assume that is QA. It's not. Moreover they never listen to QA nor respect them as people. QA are usually the best people to know what's what when the rubber meets the road. They know where the bodies are buried. They often understand and have a pulse on customers and usage better than product managers. They know the ins-and-outs of how various features interact and how a product as a whole works better than silo'd developers. Historically they were effectively the only "product owners" that could certify a release before it went out. They would coordinate with the right people to ensure all technical and non-technical deliverables and dependencies were met before releases. T hey would be the best approximate and power users. They often maintained test infrastructures on which deployments could be tested. In fact, they were the origins of automation or DevOps as a "thing" because they are the ones who saw all the friction points daily giving rise to CI/CD. Often times nobody listened because they were concerned with features. QA has always been about investing resources within an organization to improve it. In effect, building things for within it to improve it. Now that we have gotten the message to some degree of optimizing some manual pain points -- we kick to the curb those who got us there without any regard to the value they provided -- and instead decide to push to prod and test on production to piss off customers even more. If you ask yourself -- would you feel comfortable driving a car that was built and tested with automation alone? What about a space shuttle or an airplane? Or about medications?Or would you prefer that a human test drivers or test pilots put those products through their paces before signing off on them? -- it might drive it home better. But then again software industry has devolved since the tech bro + VC culture pretty much ate it as they chased those sweet $$$

mynameisnoone 3y ago

It's not so much a Shakespearean "to have a QA team or not ...", but responsibility and accountability. SWEs must produce unit testing because they know the code best. Dumping this responsibility onto QA is slow, evil, and wrong like quality control only vs. QA+QC. QA teams must have the authority to ensure complicated code gets unit testing code coverage. QA teams should provide partial and full-up integration tools and testing with the support of SWEs. QA teams must have stop-the-assembly-line authority to ensure quality and testing requirements are met. QA teams (or tools teams that support multiple QA teams) must make testing faster and more efficient. There ought to be a suite of smoke tests such that untested code cannot be committed to the "production" branch, whatever that looks like, except in rare extreme emergencies. All production-ready commits should be squashed, signed-off by another SWE, and have a test plan. Test plans should be auto-generated, wherever possible. Tests combined with test infrastructure should be able to be added to auto-bisect/blame breakage Which tests must run and pass to accept a production proposed diff should be auto-minimized to those that touch particular area(s) and their precise-as-possible-but-still-correct dependencies. Other areas that must not be neglected and shoveled onto SWEs: product management, UAT, UX, and operations.

xtracto 3y ago
I am a CTO and have always tried to have the QA team to be part of the Product Office (CPO) . It tends to keep us honest and ultimately it aligns the decision of churning out new features and building crap in the same team/person. Now, don't get me wrong, my teams always do unit and integration testing, along with automation of those two. Devs are responsible for the quality of their work. But ultimately it os the product team, with input from their QA team the ones deciding if a new feature is ready for release as it is, or needs more polish

physicsguy 3y ago

I personally think that having worked in both environments, the biggest issue is companies missing that triage function for bugs, etc. from people who can turn a customer reported bug into a step by step guide for reproducing things. When I worked in the simulation space we used to get models sent in by customers where a convergence problem or crash would occur in 12 hours of running on a 128 core machine. Those were impossible as a developer to work with in debug mode which made the runtime even longer, so they needed someone to identify the cause of the problem and distill it down to a much smaller model where the bug could be replicated. The QA team in that were really application engineers and were subject matter experts, and they were absolutely invaluable.

bigEnotation 3y ago

>>> slowest part of software delivery is testing In my experience the slowest part has been marking a feature as done. I loved working at places with QA. I could assign tickets to QA once the PR was up. Now I gotta build in that I’ll be bumping PRs for review for approximately 30-50% of the time I’m working on a feature.

gluteart 3y ago

I joined my current company just months before they started to cut down small QA team we had then. QA Automation was supposed to be the answer. 1.5 years forward - product quality dropped, automation for the client side application barely exists and even those parts that is covered by it prone to bugs. I tried to draw attention to the fact that at least some manual QA is needed, but even after obvious fail (some people lost their job) managers are adamant. Automation, 'special bug-hunting projects', 'we should concentrate on code quality' lectures, all-hands testing - anything, instead of very obvious solution to get QA team back. Development time is up, regression is often, communication became harder. The only QA who still works in the company (now in a different role) became invaluable, because he is one of the very few people who deeply understands the product as a whole and knows all the services we work with. I can't think of another example of so very obvious mistake and solution to it, that's being ignored so relentless.