18 ms·

Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with

Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with government systems, policies and personnel - things that government already knows how to do? "Because we don't have the expertise to do software development." The comments in this thread, which I fully agree with by the way, just cement my belief that if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house.

didibus 6y ago
Exactly, that's why I turn down companies that don't take a tech first approach, if you rely on tech, you ARE a tech company. Clearly the government doesn't see itself as a tech venture of any kind, but tech fails in silo. Tech first is the only way tech can succeed at delivering value to your business. Otherwise, go back to traditional processes, stick to SharePoint, email, excel, and manual logistics. And ya, this is true of the government as well, if it wants to modernize and leverage software innovations it'll need to start approaching things with a tech first mindset.
nataz 6y ago
Ha, I'm currently a federal government program manager, so I am thinking about it from the government's side right now. I've also been a consultant cog in a mega-corp and run my own small consulting business, so I've seen it from both sides. Tech folks underestimate non tech challenges (basically anything outside of their domain), and clients don't actually know what they want. It's an age old problem in lots of industries.
- rorykoehler 6y ago
  I’ve been working in enterprise/government sector for a few years now and my observation is that they need to focus on better systems design. Everything is too convoluted. Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear is where most of the money is wasted.
  - eecc 6y ago
    Oh, my last disappointment of the week was being lectured via 2 screenfuls of Confluence with a “it’s more complex that what you make it” exposure of 2-3 layers of encrusted legacies. The PO was flexing and glowing how their managed to build into the new product this synthesis of bad designs. Which means entire departments get a list of rotating passwords by email every couple weeks. The network of ossified legacies just grew one more sprout.
  - Aeolun 6y ago
    > Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear “We need a new system, but it has to function exactly like the old one.” Including any bugs and craziness.
    - GVIrish 6y ago
      I've wasted far too much time in my career doing exactly this for federal government customers. My favorite is when someone insists a feature is critical and must be replicated, but when I dig into it I find it *never* actually worked and as such, no one was using it.
    - fallingfrog 6y ago
      I’ve seen several new systems that actually were the old system, with a screen scraper and a web interface slapped on top.
- imperfectcats 6y ago
  Made worse when you have lots of money and even more stakeholders. Perhaps worst of all, when failure has no true cost, avoiding failure matters far less. The CDC won't go broke because they blew through $44 Mil. Most every other organisation would. That's one of the challenges of government, the lack of creative destruction.
- seebetter 6y ago
  Lol, I worked in hardware development for 8 years and politics for a few and now government research. 100% of problem is the core underlying philosophical concept of the theory of administration and bureaucracy. It’s not a broken system, it’s a broken concept.
- TheColorYellow 6y ago
  I work in a similar sector but have a strong belief that the software delivery can and should become a capability that occurs within the government. They should be diligent in what capabilities they choose to support. And ruthless in continually developing and pivoting these capabilities. This is true of any large organization or enterprise that operates in a software enabled domain, so yes, most industries. The idea that software development is too difficult and should only be contracted out to fleety consultants is one that needs to be properly put to rest. And to be clear, I'm not saying that consultants have no place. But for the type of technology and delivery as required by the project in the OP - there's no excuse.
- allie1 6y ago
  Would you say the price is justified?
- oivey 6y ago
  Domain expertise is important, but I really struggle to see the 10s of millions of domain expertise at play here.
fakedang 6y ago
Exactly. The UK in-housed their entire website IT team, and it's one of the best set of government websites I've seen worldwide, UX-wise and performance-wise.
- j45 6y ago
  If I recall correctly the UK digital govt is leveraging Camunda, which has some mindbendingly good pieces. Camunda is not as well known in North America, it will be interesting to see what emerges in the digital transformation/rpa space.
- onion2k 6y ago
  There are a lot of Accenture consultants working in the Digital Service.
- crimsoneer 6y ago
  GDS is a shadow of its former self now though
  - fakedang 6y ago
    Tbh the UK is a shadow of its former self now, as are all things in it.
ethbr0 6y ago
This is pretty much what I've learned doing 5+ years of big enterprise consulting (though not government). If the hardest / longest part of any project requires domain knowledge and personal relationships, then why are you outsourcing that part? I don't care if it's not a core competency / something you can do. Make it one.
- eaton 6y ago
  One of the biggest challenges for large orgs is that the knowledge they need to do a given thing is almost always present inside the organization — but it's fragmented and scattered across different teams, roles, and balkanized administrative boundaries such that the big picture is almost never available. A high-quality consulting partner works to build the relationships and internal lines of communication that are necessary for the client organization to do more of the work — maybe all of it! — in the future. A cash-grab consultancy will just make themselves the organization's "glue" and bill until the money runs dry.
  - acdha 6y ago
    That last dynamic you described is exactly what I saw at a very well known university: upper management knew they weren’t getting the accurate details through the middle management filter, but shied away from fixing the cultural problems. Instead, a major consulting company was charging a considerable amount per hour to have 24 year olds with expensive suits interview the actual workers, summarize what they’d been telling their management for years, and present it directly to the senior level. Fixing the system would have saved millions but required fights they weren’t constitutionally well suited for.
    - ethbr0 6y ago
      Absolutely, but this is essentially saying "We're okay paying a 2-10% tax on our revenue, forever." I think that's the reason you tend to see these unhealthy dynamics in extremely conservative, quasi-monopoly industries (government, financial, healthcare), where there's no real question of if they'll be a going concern. And no real rewards / attempts to outperform at the cost of increased risk.
pjscott 6y ago
Does the government, in fact, know how to do those things? How would we know if they did or didn't? I notice that they spend oddly large amounts of money on anything for which we know what the cost roughly should be -- e.g. CDC web sites -- and don't seem to do so very quickly or very well. Should we assume that they're a lot more competent at things we know less about?
- angry_octet 6y ago
  Agencies are less likely to be competent in non core areas. This should not be surprising. There is no career path for you doing software in an agency like the CDC run by bio science people. Same for the Air Force, which is run by fast jet pilots, the Congress by lawyers and liars, Commerce by thieves, etc. Even banks and trading firms, who live and die by software, are run by accountants and economists. Unless an organization recognizes the core importance of software and carves out a specialisation stream and C level ownership of it, they rarely achieve more than brief or isolated competence. This is the reason why SV is so appealing -- nowhere else do SWEs hold sway over entire companies and industries. Govt can do excellent software if it becomes a priority, they could pay competitive wages (there are special pay scales for doctors and lawyers), but they choose not to because of risk aversion from politicians, who mistakenly believe they can outsource the risk. For something like this, paying Eventbrite to do it wouldn't have been a bad place to start. But the fat cat consulting firms never have domain expertise beyond getting Govt contracts.
- GVIrish 6y ago
  There's a couple of reasons government IT projects turn out this way. In my experience, a lot of the time it's not so much that the contractor is trying to make money by prolonging the problem. It's that the customer doesn't know how to be a competent product owner that prioritizes and scopes things properly. Then, frequently the software being designed ends up replicating broken/inefficient business processes, so it ends up being difficult to make cleans and efficient software. This becomes worse when there's legacy software in the mix where the customer wants the contractor to port over all of the clunky functionality of the old app. Which of course isn't documented in the first place, so you've got to perform software archaeology to figure out what it did. Some other show stoppers I've seen are when key stakeholders act as gatekeepers and prevent developers from talking to SMEs directly, so you have to build your requirements based on what the person with the 10,000 foot view knows. So of course you get incomplete and sometimes completely wrong requirements. And you might not find out how wrong they are until after you've spent a lot of time building things Just those dynamics alone can add an enormous amount of cost or outright doom a project. I could probably fill a blog with all of the dysfunctional stuff I've seen in federal IT, but suffice it to say that these factors can easily balloon costs and timelines to a comical degree. This $44 million failure is actually small compared to some other government IT failures like the billion dollar Air Force accounting system.
tomrod 6y ago
They have started. https://www.usds.gov/ https://www.usds.gov/
franklampard 6y ago
corruption?
mellosouls 6y ago
There is often the expertise in house, but over years the internal teams get terribly under-funded and the poor service that results is then used to justify spending vastly more amounts on external suppliers. This always comes down to piss-poor and craven senior management decisions.
Aeolun 6y ago
> if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house. This is basically true for any organisation.
chrisweekly 6y ago
The US has a CTO, so there seems to be some structure to support this notion, right?
- aweiland 6y ago
  It does. However every piece of the government has their own as well who don't seem at all beholden to it.
sprafa 6y ago
the UK has done exactly this - to make all Government Digital Services in house. It's been a blockbuster success in my opinion - they have some of the best UX/UI in the world and their websites are ultralight and downgrade well. There are a few issues with authentication, but there is support. Everything that's related to dealing with the UK government btw is all in ONE website, where everything is explained to you in Plain English - gov.uk Coming from a country where this doesn't exist, I was blown away by every interaction I've had with the system. It's simply amazing.
- cyberpunk 6y ago
  Every single GDS project I’ve been on is 90% freelancers all via a small handful of companies — it’s not really “in-house” but at least it’s better than absolutely everything going to capgemini, Fujitsu, NTT, ATOS and the like, sure.
  - sprafa 6y ago
    Did not know that, still the QC of their work and overall integration of services seems world-beating to me. I had never seen anything like it. Their UX documents and implementation are for my money some of the finest in the industry.
- williamdclt 6y ago
  As a foreigner living in the UK, I concur. All .gov.uk are _extremely_ functional. Applying for the pre-settled status (cos Brexit) was actually an enjoyable experience! I had no idea what was going to happen when I started the process, and it was all done in 30min top. It made me _scan my passport with my smartphone_ while flashing weird colors and it worked flowlessly, that absolutely blew my mind. I've actually looked into getting a job there even though I'm from a start-up background
  - sprafa 6y ago
    Yes that was some of their best work. I believe the flashing lights allow for some sort of 3D reconstruction of your face? And scanning the passport with NFC! One of them is high-tech DYI solution, the other one is super simple yet not used at all by any other govt service I've interacted with.
patentatt 6y ago
The sad, true reason why they don’t is that government programmer jobs will pay 1/2 - 1/3 what a private sector job will pay in the DMV (DC, MD, VA). And the ironic part is that the reason private sector pay is so high is largely attributable to these big federal contracts being handed out. But as things are, the federal government cannot hire enough qualified people in the location (by DC) that they want to.
umanwizard 6y ago
I’ve heard the main reason is that it’s impossible for the government to pay market rate for high-end software engineers, because of fixed pay bands.
Demiurge 6y ago
I’m going to be discrete but I know of a Deloitte project where they are allowed to use AWS Fargate, but the gov’t side is limited to IBM mainframes. Why are they so limited? Who knows, but it’s a deep question, that would require total reinvention. Perhaps, in the end, it might not even be worth the risk because the gov’t is so essential. They can’t move fast and break things. But it does look insane as a result.
- treis 6y ago
  FEDRAMP requirements and auditing are difficult and expensive to meet. I've seen the cost of these in bids. We said something like $20k per month for what would be a $50 per month EC2 server. It was a shocking thing to see. I'm guessing Deloitte is charging six figures a month to host this thing.
atlgator 6y ago
The problem is you can't fire Government employees. The impact of hiring specialized staff that don't perform is very high.
WhompingWindows 6y ago
I worked for the government and it was pretty damn hard to get further training. We had no funds for conferences, classes, travel, or any such. All the improvement was down to me, what I could learn about R/stats/STATA from forums, from colleagues, and from the 1500 member analyst email list. There were all sorts of forums/libraries "within" the government system, however their search functionality was garbage and they didn't have enough users to make it a useful forum for most topics. Sigh!

didibus 6y ago

Exactly, that's why I turn down companies that don't take a tech first approach, if you rely on tech, you ARE a tech company. Clearly the government doesn't see itself as a tech venture of any kind, but tech fails in silo. Tech first is the only way tech can succeed at delivering value to your business. Otherwise, go back to traditional processes, stick to SharePoint, email, excel, and manual logistics. And ya, this is true of the government as well, if it wants to modernize and leverage software innovations it'll need to start approaching things with a tech first mindset.

nataz 6y ago

Ha, I'm currently a federal government program manager, so I am thinking about it from the government's side right now. I've also been a consultant cog in a mega-corp and run my own small consulting business, so I've seen it from both sides. Tech folks underestimate non tech challenges (basically anything outside of their domain), and clients don't actually know what they want. It's an age old problem in lots of industries.

rorykoehler 6y ago
I’ve been working in enterprise/government sector for a few years now and my observation is that they need to focus on better systems design. Everything is too convoluted. Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear is where most of the money is wasted.
- eecc 6y ago
  Oh, my last disappointment of the week was being lectured via 2 screenfuls of Confluence with a “it’s more complex that what you make it” exposure of 2-3 layers of encrusted legacies. The PO was flexing and glowing how their managed to build into the new product this synthesis of bad designs. Which means entire departments get a list of rotating passwords by email every couple weeks. The network of ossified legacies just grew one more sprout.
- Aeolun 6y ago
  > Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear “We need a new system, but it has to function exactly like the old one.” Including any bugs and craziness.
  - GVIrish 6y ago
    I've wasted far too much time in my career doing exactly this for federal government customers. My favorite is when someone insists a feature is critical and must be replicated, but when I dig into it I find it *never* actually worked and as such, no one was using it.
  - fallingfrog 6y ago
    I’ve seen several new systems that actually were the old system, with a screen scraper and a web interface slapped on top.
imperfectcats 6y ago
Made worse when you have lots of money and even more stakeholders. Perhaps worst of all, when failure has no true cost, avoiding failure matters far less. The CDC won't go broke because they blew through $44 Mil. Most every other organisation would. That's one of the challenges of government, the lack of creative destruction.
seebetter 6y ago
Lol, I worked in hardware development for 8 years and politics for a few and now government research. 100% of problem is the core underlying philosophical concept of the theory of administration and bureaucracy. It’s not a broken system, it’s a broken concept.
TheColorYellow 6y ago
I work in a similar sector but have a strong belief that the software delivery can and should become a capability that occurs within the government. They should be diligent in what capabilities they choose to support. And ruthless in continually developing and pivoting these capabilities. This is true of any large organization or enterprise that operates in a software enabled domain, so yes, most industries. The idea that software development is too difficult and should only be contracted out to fleety consultants is one that needs to be properly put to rest. And to be clear, I'm not saying that consultants have no place. But for the type of technology and delivery as required by the project in the OP - there's no excuse.
allie1 6y ago
Would you say the price is justified?
oivey 6y ago
Domain expertise is important, but I really struggle to see the 10s of millions of domain expertise at play here.

fakedang 6y ago

Exactly. The UK in-housed their entire website IT team, and it's one of the best set of government websites I've seen worldwide, UX-wise and performance-wise.

j45 6y ago
If I recall correctly the UK digital govt is leveraging Camunda, which has some mindbendingly good pieces. Camunda is not as well known in North America, it will be interesting to see what emerges in the digital transformation/rpa space.
onion2k 6y ago
There are a lot of Accenture consultants working in the Digital Service.
crimsoneer 6y ago
GDS is a shadow of its former self now though
- fakedang 6y ago
  Tbh the UK is a shadow of its former self now, as are all things in it.

ethbr0 6y ago

This is pretty much what I've learned doing 5+ years of big enterprise consulting (though not government). If the hardest / longest part of any project requires domain knowledge and personal relationships, then why are you outsourcing that part? I don't care if it's not a core competency / something you can do. Make it one.

eaton 6y ago
One of the biggest challenges for large orgs is that the knowledge they need to do a given thing is almost always present inside the organization — but it's fragmented and scattered across different teams, roles, and balkanized administrative boundaries such that the big picture is almost never available. A high-quality consulting partner works to build the relationships and internal lines of communication that are necessary for the client organization to do more of the work — maybe all of it! — in the future. A cash-grab consultancy will just make themselves the organization's "glue" and bill until the money runs dry.
- acdha 6y ago
  That last dynamic you described is exactly what I saw at a very well known university: upper management knew they weren’t getting the accurate details through the middle management filter, but shied away from fixing the cultural problems. Instead, a major consulting company was charging a considerable amount per hour to have 24 year olds with expensive suits interview the actual workers, summarize what they’d been telling their management for years, and present it directly to the senior level. Fixing the system would have saved millions but required fights they weren’t constitutionally well suited for.
  - ethbr0 6y ago
    Absolutely, but this is essentially saying "We're okay paying a 2-10% tax on our revenue, forever." I think that's the reason you tend to see these unhealthy dynamics in extremely conservative, quasi-monopoly industries (government, financial, healthcare), where there's no real question of if they'll be a going concern. And no real rewards / attempts to outperform at the cost of increased risk.

pjscott 6y ago

Does the government, in fact, know how to do those things? How would we know if they did or didn't? I notice that they spend oddly large amounts of money on anything for which we know what the cost roughly should be -- e.g. CDC web sites -- and don't seem to do so very quickly or very well. Should we assume that they're a lot more competent at things we know less about?

angry_octet 6y ago
Agencies are less likely to be competent in non core areas. This should not be surprising. There is no career path for you doing software in an agency like the CDC run by bio science people. Same for the Air Force, which is run by fast jet pilots, the Congress by lawyers and liars, Commerce by thieves, etc. Even banks and trading firms, who live and die by software, are run by accountants and economists. Unless an organization recognizes the core importance of software and carves out a specialisation stream and C level ownership of it, they rarely achieve more than brief or isolated competence. This is the reason why SV is so appealing -- nowhere else do SWEs hold sway over entire companies and industries. Govt can do excellent software if it becomes a priority, they could pay competitive wages (there are special pay scales for doctors and lawyers), but they choose not to because of risk aversion from politicians, who mistakenly believe they can outsource the risk. For something like this, paying Eventbrite to do it wouldn't have been a bad place to start. But the fat cat consulting firms never have domain expertise beyond getting Govt contracts.
GVIrish 6y ago
There's a couple of reasons government IT projects turn out this way. In my experience, a lot of the time it's not so much that the contractor is trying to make money by prolonging the problem. It's that the customer doesn't know how to be a competent product owner that prioritizes and scopes things properly. Then, frequently the software being designed ends up replicating broken/inefficient business processes, so it ends up being difficult to make cleans and efficient software. This becomes worse when there's legacy software in the mix where the customer wants the contractor to port over all of the clunky functionality of the old app. Which of course isn't documented in the first place, so you've got to perform software archaeology to figure out what it did. Some other show stoppers I've seen are when key stakeholders act as gatekeepers and prevent developers from talking to SMEs directly, so you have to build your requirements based on what the person with the 10,000 foot view knows. So of course you get incomplete and sometimes completely wrong requirements. And you might not find out how wrong they are until after you've spent a lot of time building things Just those dynamics alone can add an enormous amount of cost or outright doom a project. I could probably fill a blog with all of the dysfunctional stuff I've seen in federal IT, but suffice it to say that these factors can easily balloon costs and timelines to a comical degree. This $44 million failure is actually small compared to some other government IT failures like the billion dollar Air Force accounting system.

tomrod 6y ago

They have started. https://www.usds.gov/ https://www.usds.gov/

franklampard 6y ago

corruption?

mellosouls 6y ago

There is often the expertise in house, but over years the internal teams get terribly under-funded and the poor service that results is then used to justify spending vastly more amounts on external suppliers. This always comes down to piss-poor and craven senior management decisions.

Aeolun 6y ago

> if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house. This is basically true for any organisation.

chrisweekly 6y ago

The US has a CTO, so there seems to be some structure to support this notion, right?

aweiland 6y ago
It does. However every piece of the government has their own as well who don't seem at all beholden to it.

sprafa 6y ago

the UK has done exactly this - to make all Government Digital Services in house. It's been a blockbuster success in my opinion - they have some of the best UX/UI in the world and their websites are ultralight and downgrade well. There are a few issues with authentication, but there is support. Everything that's related to dealing with the UK government btw is all in ONE website, where everything is explained to you in Plain English - gov.uk Coming from a country where this doesn't exist, I was blown away by every interaction I've had with the system. It's simply amazing.

cyberpunk 6y ago
Every single GDS project I’ve been on is 90% freelancers all via a small handful of companies — it’s not really “in-house” but at least it’s better than absolutely everything going to capgemini, Fujitsu, NTT, ATOS and the like, sure.
- sprafa 6y ago
  Did not know that, still the QC of their work and overall integration of services seems world-beating to me. I had never seen anything like it. Their UX documents and implementation are for my money some of the finest in the industry.
williamdclt 6y ago
As a foreigner living in the UK, I concur. All .gov.uk are _extremely_ functional. Applying for the pre-settled status (cos Brexit) was actually an enjoyable experience! I had no idea what was going to happen when I started the process, and it was all done in 30min top. It made me _scan my passport with my smartphone_ while flashing weird colors and it worked flowlessly, that absolutely blew my mind. I've actually looked into getting a job there even though I'm from a start-up background
- sprafa 6y ago
  Yes that was some of their best work. I believe the flashing lights allow for some sort of 3D reconstruction of your face? And scanning the passport with NFC! One of them is high-tech DYI solution, the other one is super simple yet not used at all by any other govt service I've interacted with.

patentatt 6y ago

The sad, true reason why they don’t is that government programmer jobs will pay 1/2 - 1/3 what a private sector job will pay in the DMV (DC, MD, VA). And the ironic part is that the reason private sector pay is so high is largely attributable to these big federal contracts being handed out. But as things are, the federal government cannot hire enough qualified people in the location (by DC) that they want to.

umanwizard 6y ago

I’ve heard the main reason is that it’s impossible for the government to pay market rate for high-end software engineers, because of fixed pay bands.

Demiurge 6y ago

I’m going to be discrete but I know of a Deloitte project where they are allowed to use AWS Fargate, but the gov’t side is limited to IBM mainframes. Why are they so limited? Who knows, but it’s a deep question, that would require total reinvention. Perhaps, in the end, it might not even be worth the risk because the gov’t is so essential. They can’t move fast and break things. But it does look insane as a result.

treis 6y ago
FEDRAMP requirements and auditing are difficult and expensive to meet. I've seen the cost of these in bids. We said something like $20k per month for what would be a $50 per month EC2 server. It was a shocking thing to see. I'm guessing Deloitte is charging six figures a month to host this thing.

atlgator 6y ago

The problem is you can't fire Government employees. The impact of hiring specialized staff that don't perform is very high.

WhompingWindows 6y ago

I worked for the government and it was pretty damn hard to get further training. We had no funds for conferences, classes, travel, or any such. All the improvement was down to me, what I could learn about R/stats/STATA from forums, from colleagues, and from the 1500 member analyst email list. There were all sorts of forums/libraries "within" the government system, however their search functionality was garbage and they didn't have enough users to make it a useful forum for most topics. Sigh!