6 ms·
In a lot of industrial sites software security is a joke. Embedded systems tend to use very old, well proven technology, which in itself isn't a problem, it fit
by oilman 7y ago
In a lot of industrial sites software security is a joke. Embedded systems tend to use very old, well proven technology, which in itself isn't a problem, it fits the market well, but the side effect is that security isn't always properly considered as it wasn't a concern when the software/hardware was developed.
I was involved in a project a few years ago delivering a series of monitoring systems running Windows XP to a brand new 700 million dollar oil rig. This was at the request of the client, they had software they needed that would only run on Windows XP. They had a fit when we had trouble sourcing Windows XP licenses. The expectation is that these systems will have a 20 - 30 year life.
It used to be that keeping every air gapped was enough, but organizations want easier monitoring, so more systems are being networked in an ad-hoc way without a lot of thought about security.
I expect we are going to see more things like this happening in the future until we start taking security in systems / embedded space more seriously. And even then there will be exploits of older systems for years afterwords since the replacement cycle is so long.
I wonder what a secure embedded system even looks like when I think about it. The environment isn't suitable to the kind of continuous patching that is done in the web world, but exploits will be found and dependencies will need to be updated. How do you square keeping things up to date with stringent testing requirements in systems that can kill people. Many of these systems / plants are unique, there is only one plant like it in the world, so testing becomes very hard.
- losteric 7y agoHow are mechanical components tested against requirements in critical systems? What is the process for changing those components, like upgrading pipes to a fancy new composite? In my mind, we'll start treating silicon the same way... formal verification, rigorous real-world testing, trusted suppliers, and an expectation that change is slow, expensive, and risky.
- jonawesomegreen 7y agoSeems like there is a huge opportunity here for a startup that can navigate the industry and manage to solve some of these problems. There are huge players in the space, but from what I've seen they aren't solving these problems very effectively.
- anitil 7y agoYou're right, but my experience suggests that anything broadly in the 'manufacturing' field is treated as a cost center, and not something people want to spend money on. In most places a rubber stamp of 'yep, definitely secure because we have a password (over http, oh by the way everyone uses the default which is in the manual)' is good enough. Then when something goes wrong there's a general shrug and they change the password.
- FuckOffNeemo 7y agoSeconding what anitil said above/below me. I've friends who work as instrument technicians/engineers on embedded systems in manufacturing and have worked in Europe, UK and Australia in food manufacturing, mechanical, water collection and water processing. The one constant I've heard is that all of their hardware is almost or is out of support, when it breaks, they expect band aid fixes and ironically none or very little of them can accept any downtime. There's no hardware redundancy for their production lines and when anything breaks, it's all hands on deck. Yet there's no funding going back into the production lines to pro-actively repair or minimise their risks. Manufacturing it seems to me is 100% a reactive industry. The industries above work on incredibly small margins of profit and the sheer expense to outfit and refit these aging, decrepit (but still working!) production lines are quite honestly, massive. These manufacturers won't invest in these engineering faults (whether it's security or production focused) until they've been fucked. NB: This might be with the exception of Lego, my cousin who got employed by Lego after finishing his masters in Industrial Design & CS, and after reading and watching some articles on Lego. I'm convinced Lego's margins are a lot larger than most. They might be the closest thing to a FAANG company when it comes to investment in phu7sical engineering and manufacturing.
- pmorici 7y agoHow are these places insured against industrial accidents? I wonder if the way to get your foot in the door would be to partner with an insurer who would provide discounts to the client if they installed security technology meeting a certain standard. You could come up with some kind of bump on the wire type Linux device that proxied access to the old insecure system bringing them up to the standard. Then sell it as a return on investment through savings on industrial accident insurance premiums.
- HeWhoLurksLate 7y agoFrom what I've heard, one of the problems here is that Windows 10's support of parallel ports (widely used in automation) is really bad, and there's no way that live updates are going to be acceptable on something that needs to be running 24/7. What ways are there to properly air-gap something like this?
- varjag 7y agoThe solution of course is to not use Windows, without ifs and buts. It was a bad idea coming to its conclusion.
- ronsor 7y agoThat's always the solution, even if it isn't a realistic one.
- an_account_name 7y agoPlenty of software runs on platforms other than windows. Plenty of industrial control software, too.
- coldacid 7y agoAnd of course once nobody's using Windows, attackers will pivot to target whatever everyone's switched to in its place. And they'll find holes. They always do.
- varjag 7y agoWe're talking about the difference between barely usable and unusable here. No system in the world has ever been slimefest of the scale of Windows. Even those which are deployed more than Windows.
- thesimp 7y agoIt is much to easy to blame Windows for everything. If this is another TRITON level attack then it did not matter what operating system is used, the attackers would have come in anyway. With enough resources you can break anything. Check out the DEFCON 2018 talk "Through the Eyes of the Attacker" to see to what lengths the TRITON attackers went. These guys were dumping eeproms from boards running obscure MIPS processors and were looking at raw ethernet packets in Wireshark and flipping bits in the packets to see what would happen. That their command&control was running on Windows was just coincidence.
- reaperducer 7y agoAlso surprising is how many of these old systems are hooked up to dial-up modems with ZERO login security. Not even a password.
- dv_dt 7y agoI'll never understand selecting windows for any embedded system that is expected to have a 10+ year life. If you look at their history, they frequently blow up the development environment every 5-6(?) years. Yes they'll provide support, but it comes at being increasingly held hostage to whatever level of long term support costs are in vogue at the time. For a 20-30 year support life, that's a guaranteed headache.
- snowwindwaves 7y agoYou don't install any updates or change anything once the machine is setup and performing its duty. If the hardware fails you replace it with similar hardware and install the original software from CD again. I see windows NT on ships, windows 2000 in power stations, etc.
- dv_dt 7y agoWeren't failing legacy computers on bridge systems one of the contributors to the Navy destroyer crashes?
- snowwindwaves 7y agoImproper maintenance of worn out components can cause a failure resulting in a crash, be they computer components, electrical components, or mechanical components.
- tsukikage 7y agoLinux doesn't necessarily help here. Ubuntu LTS lifetime is 3-5 years, others are similar; outside this, distro maintainers think nothing of just dropping packages from the distro that the deployment may be relying on, so upgrading is not simple.
- cesarb 7y agoThe difference is that with Linux you can self-support, or pay for someone else to do it, since the whole source code is available to everyone and the licence allows it. Also, a better comparison would be with RHEL, which has a lifetime of 10 years or more.