11 ms·

The gigantic and unregulated power plants in the cloud

456 points by ahubert 2y ago

deleted 2y ago
[deleted]
samstave 2y ago
I posted this question to HN 7 months ago, more around DataCenters: >In the increasingly interconnected global economy, the reliance on Cloud Services raises questions about the national security implications of data centers. As these critical economic infrastructure sites, often strategically located underground, underwater, or in remote-cold locales, play a pivotal role, considerations arise regarding the role of military forces in safeguarding their security. While physical security measures and location obscurity provide some protection, the integration of AI into various aspects of daily life and the pervasive influence of cloud-based technologies on devices, as evident in CES GPT-enabled products, further accentuates the importance of these infrastructure sites. >Notably, instances such as the seizure of a college thesis mapping communication lines in the U.S. underscore the sensitivity of disclosing key communications infrastructure. >Companies like AWS, running data centers for the Department of Defense (DoD) and Intelligence Community (IC), demonstrate close collaboration between private entities and defense agencies. The question remains: are major cloud service providers actively involved in a national security strategy to protect the private internet infrastructure that underpins the global economy, or does the responsibility solely rest with individual companies? --- And then I posted this, based on an HNers post about mapping out Nuclear Power Plants: https://news.ycombinator.com/item?id=41189056 https://news.ycombinator.com/item?id=41189056 [We can easily map the infrastructure of the cloud and AI -- and their supply chains - and these are increasingly of National Security Concern:] ((Not to mention the actual powerplants being built to exclusively provide datacenter power)) Now, if we add the layers of the SubmarinCableMap [0] DataCenterMap [1] - and we begin to track shipments And https://i.imgur.com/zO0yz6J.png https://i.imgur.com/zO0yz6J.png -- Left is nuke, top = cables, bottom = datacenters. I went to ImportYeti to look into the NVIDIA shipments: https://i.imgur.com/k9018EC.png https://i.imgur.com/k9018EC.png And you look at the suppliers that are coming from Taiwan, such as the water-coolers and power cables to sus out where they may be shipping to, https://i.imgur.com/B5iWFQ1.png https://i.imgur.com/B5iWFQ1.png -- but instead, it would be better to find shipping lables for datacenters that are receiving containers from Taiwain, and the same suppliers as NVIDIA for things such as power cables. While the free data is out of date on ImportYeti - it gives a good supply line idea for NVIDIA... with the goal to find out which datacenters that are getting such shipments, you can begin to measure the footprint of AI as it grows, and which nuke plants they are likely powered from. Then, looking into whatever reporting one may access for the consumption/util of the nuke's capacity in various regions, we can estimate the power footprint of growing Global Compute. DataCenterNews and all sorts of datasets are available - and now the ability to create this crawler/tracker is likely full implementable https://i.imgur.com/gsM75dz.png https://i.imgur.com/gsM75dz.png https://i.imgur.com/a7nGGKh.png https://i.imgur.com/a7nGGKh.png [0] https://www.submarinecablemap.com/ https://www.submarinecablemap.com/ [1] https://www.datacentermap.com/ https://www.datacentermap.com/
dataflow 2y ago
> It’s also possible that the manufacturer gets hacked, and subsequently sends out attacker controlled and wrong software updates to the inverters, with possibly dire consequences. > There are also people that claim that the many Chinese companies managing our power panels for us might intentionally want to harm us. Who knows. Wait, seriously? The European power system relies on Chinese companies not messing it up remotely? And the debate is over whether the companies will stay nice? For heaven's sake, isn't it obvious that during a war the Chinese government can force them to just destroy the continent's power system remotely? How is this not seen as a extreme continental security risk?
- formerly_proven 2y ago
  Same continent that bought energy for decades from its strategic enemy. Coincidence? Probably not. Boundless naivete and corruption? Also yes.
  - ragebol 2y ago
    Russia wasn't an enemy for a while. The belief was that engaging with them would ensure they wouldn't be an enemy again. That failed. Germany was an enemy once as well
- amelius 2y ago
  They already can by simply turning open some power mosfets in their fleet of EVs.
  - bilbo0s 2y ago
    Yeah. I'm not sure everyone is really thinking clearly here. Don't get me wrong, they should get rid of this practice of cloud monitoring. A consumer should be able to access monitoring over the internet without an intermediary. They should, of course, be allowed to contract with an intermediary if that is their desire. But the security argument? Yeah, that ship has sailed. Total war, means total war. Your power grid, your internet, your communications, and your fossil fuel deliveries will all see material disruption. I wouldn't count on being able to stop those disruptions by banning a few web sites. (And frankly, during total war, those disruptions would be the least of your problems in any case.) Best bet for places like Europe, China, the US and Russia is, just don't do total war with each other. If you choose to do it anyway, then you can see what you can expect from that in the documents filed under "Play stupid games, win stupid prizes."
yetihehe 2y ago
>The owner of the panels and inverters can meanwhile establish a connection with that manufacturer using an app or website, and via the manufacturer see how their own panels are doing > It wasn’t necessary from a technical standpoint to let everything run through the manufacturer’s servers, but it was chosen to do it this way. (emphasis from article) I'm working on IoT cloud system. It was chosen to be done this way because netither consumers nor installers have any expertise whatsoever to setup their own network or any devices to be acessible from outside (and they want their panels to be accessible when they are outside their home). I can do it, most readers of HN could do it, but typical consumer or installer can't. Sad but true.
- Nextgrid 2y ago
  The cloud can operate as a dumb TURN relay relaying E2E-encrypted traffic. Then the worst the cloud can do is deny service to remote management (and even then, local management would still work), but it wouldn't be able to send direct control commands to the equipment since they don't have the authentication nor encryption keys. This also makes it simpler from a programming point of view - instead of having separate cloud sync & local control protocols, you just have one local protocol and you merely tunnel it through the (dumb) cloud if you can't connect directly.
  - yetihehe 2y ago
    It could, but this requires to store historical data about usage on devices. If you store that encrypted data in cloud, then getting it to your mobile phone is super slow. If you store it in cloud, you can get historical data even if your device is dead or has 256 BYTES of memory and 1 megabit of flash storage. We have such devices, very effective at managing local municipal heating network and controlling several thermal controllers each via rs232 or rs485. Fortunately we preemptively moved everything into VPN'ed mobile network, we need special approval to touch anything on that network and can't connect without them granting access, so after EU started moving with cybersecurity this year, we are covered. > This also makes it simpler from a programming point of view - instead of having separate cloud sync & local control protocols, you just have one local protocol and you merely tunnel it through the (dumb) cloud if you can't connect directly. Having only cloud protocol is even simpler, I've done all of the above (I do backend and our firmwares).
dathos 2y ago
I live off-grid, power and water wise, and it really irked me that the monitoring coming with my inverter is only available online. Even when there is a network available the app will not work. I fixed this by getting a raspberry pi connected and reading it from there, but if I disconnect the inverter from the internet it will create a new network so now there is always an open network in the middle of nowhere with no option to disable it. I'm thinking about screwing it open and desoldering the wifi module but honestly I'll replace it in the next couple of years so I'd rather not kill myself by making a mistake.
- ansible 2y ago
  The high-voltage side should be separated from the electronics, so it shouldn't be dangerous if you are observant. It may be sufficient to just disconnect the antennas from the WiFi module, that will help prevent any network connections.
  - Nextgrid 2y ago
    Disconnecting the antenna would still have leakage at close range. Grounding the antenna might be a better option. But in practice, the dangers highlighted by the article only surface when an attacker has control of many solar plants at scale. Compromising an individual one by getting close-range physical access will be a local annoyance but wouldn't scale to a level where it can threaten the grid, so it limits the pool of potential attackers to local vandals (which can achieve their goals easier by just throwing rocks at your panels).
    - ijustlovemath 2y ago
      Without an antenna, even at close range, initial handshakes will fail or be unreliable.
  - serial_dev 2y ago
    Disclaimer, ymmw, if you have no clue about these systems (average people), you can still easily kill yourself in the process.
formerly_proven 2y ago
Most newer solar inverters can't even be set up without internet and most functions are only available with an always-on internet connection. This is also true for EU companies like SMA for example.
- grecy 2y ago
  I just installed a Fronius inverter (made in Austria) and 6.8kW of panels. The inverter itself functions perfectly fine without an internet connection, and will display instantaneous power output on the screen. I could just be content with that and look at my monthly power bill to see how much I generated and how much I used each month and never connect it to the internet. To get any kind of data logging & history from the inverter, it must be internet connected (wifi or ethernet). And all of that is through the manufacturer's website, which constantly nags me to "upgrade to pro" for some obscure feature that I'll never use.
ano-ther 2y ago
Eye opening for me. One of the arguments for renewable energy (besides emissions) has always been its potential for decentralizing power generation. Makes it more resilient, democratizes the means of production etc. This article shows that we inadvertently introduced new choke points. And of course the global security environment makes it more worrisome.
- panki27 2y ago
  Hmm, almost like what happend to the internet... the idea being "everything is decentralized", but now +80% of traffic passes through Cloudflare and over 90% of mails come from 2 providers!
  - paxys 2y ago
    Cloudflare absolutely does not control 80% of internet traffic. I have no idea where you got that number from.
    - ezfe 2y ago
      This article says 80% of known websites, which are 19% of all websites. Probably where it came from. https://w3techs.com/technologies/details/cn-cloudflare https://w3techs.com/technologies/details/cn-cloudflare
      - paxys 2y ago
        That's still the number of websites, not their traffic. A personal blog hosted on Cloudflare and google.com are not both the same.
        Aachen 2y ago
        Probably that just skews it worse though. Most people aren't cf customers but many of the big targets seek protection
- realusername 2y ago
adolph 2y ago
The short version: most consumer and business solar panels are centrally managed by a handful of companies, mostly from countries outside of Europe. In the Netherlands alone, these solar panels generate an output equivalent to at least 25 medium sized nuclear power plants. There are almost no rules or laws in Europe governing these central administrators. . . . The same thing goes for heat pumps, home batteries, and EV charging points. Seems to me that this is very similar to the situation with IoT only with higher stakes. I appreciate this article's presentation of inverter and grid trust. Beyond trusting customer inverters to do the right thing, I wonder if there is a method for safing a grid at the hardware level. Naive question: could there be a grid provider device that prevents overcurrent or incorrectly clocked cycles?
- kwhitefoot 2y ago
  The utility company fuse between the property and the 240 V distribution system should prevent overcurrent. If the frequency or phase of the inverter is wrong the inverter might die first unless the network is already down. There isn't really any practical way to prevent overvoltage though. So a rogue controller in charge of all the solar systems in a street might be able to do quite a lot of damage to consumer devices. A problem from the utility point of view is that they can no longer guarantee that the 240 V side of the distribution system is safe to work on just by tripping a breaker on either side of the distribution transformer. So all work on the 240 V distribution system has to be done with the assumption that the system is live. Eventually regulations will be updated, if necessary, to deal with large numbers of solar installations on domestic buildings.
  - cesarb 2y ago
    > The utility company fuse between the property and the 240 V distribution system should prevent overcurrent. If the frequency or phase of the inverter is wrong the inverter might die first unless the network is already down. To put it more simply: if the phase is wrong, the effect is the same as a short circuit, which fuses and circuit breakers protect against. If the frequency is wrong, the phase will become wrong after a number of cycles. > There isn't really any practical way to prevent overvoltage though. So a rogue controller in charge of all the solar systems in a street might be able to do quite a lot of damage to consumer devices. There is, it's called a surge protector or surge protective device (SPD). It converts any overvoltage above a certain level into a short to ground, which then trips the fuse or circuit breaker. It's often used as a protection against lightning-induced currents. > A problem from the utility point of view is that they can no longer guarantee that the 240 V side of the distribution system is safe to work on just by tripping a breaker on either side of the distribution transformer. So all work on the 240 V distribution system has to be done with the assumption that the system is live. From what I've seen, the utility workers usually ground the wiring when working on it (they have a special-purpose device for that). Once it's safely connected to ground, it's no longer live.
ThrowawayTestr 2y ago
If the general public knew how fragile the power grid is no body would be able to sleep at night.
- asynchronous 2y ago
  Not to be that guy, but the DOE is arguably one of the most important federal agencies in the US, and they treat the problem with the correct amount of focus, research and dedication. It’s just a very hard problem. The grid is no less secure or less resilient than it was 50 years ago, the main problem is that people are more dependent on it. Almost no one buys a personal generator before an outage happens anymore, despite it being one of the cheapest ways to get resiliency.
twoodfin 2y ago
Isn’t the right place to fix this at the junction between the plants and the grid? Regulate the grid utilities into a gateway role, and require all inverter control & telemetry traffic to pass through them. This seems likely to be more fruitful than attempting to regulate 400 Chinese panel manufacturers. What am I missing?
- itishappy 2y ago
  You're thinking about this right, just at a utility scale. The "plants" in this context are homes and businesses. The junction points between plants and grid are the inverters sold by the panel manufacturers.
WaitWaitWha 2y ago
Q: Are there no regulatory requirements for power plants of any kinds in EU, specially around cybersecurity? I do not allow any system into my environments (at home and at work) that requires a third party data connection function. There are way too many incidents where a provider, cloud or otherwise which required connection failed for various reasons. (e.g., Cisco Spark Board, Xerox ConnectKey, Google Cloud Print, WeWork's Connected devices, Lattice Egnines, MS Groove Music Pass, Shyp, Adobe Business Catalyst, Samsara, Zune, FuelBand, Anki Vector Robot, Google Stadia, Pebble) Despite this, I am very leery of regulating solar power specifically.
- afh1 2y ago
  Smartphones don't count?
  - WaitWaitWha 2y ago
    Apologies, but do not understand the question. Are you suggesting using smart phones should count in "not allowing it in"? Then yes, I try to where possible. I do not depend on a smart phone. All functionality that are operationally necessary can be done elsewhere without major delays or impact.
    - afh1 2y ago
      Interesting. How do you handle MFA, do you have a special device for that? Your bank/brokerage don't require their app?
- numpad0 2y ago
  How would one practically verify and certify cybersecurity of a product? Even payment smartcards sometimes come with non-malicious maintenance backdoors. There seem to be little to no academic theoretical basis to this whole software security thing.
  - g_p 2y ago
    Given the challenges of techniques like TLS interception (i.e. through pinning and other good security features), about the only measure I can see left is network isolation. You can set up a local network that has no WAN connectivity on it. About anything else is difficult to verify even the most basic of security properties. Certifying is another step up (although you could argue certifying is just a third party saying something passed a finite list of tests) - the real challenge is defining a meaningful certification scheme. There has been some good work towards consumer IoT device security (i.e. the 13 steps approach from the UK), that covers some of the lowest hanging fruit - https://www.gov.uk/government/publications/code-of-practice-for-consumer-iot-security/code-of-practice-for-consumer-iot-security https://www.gov.uk/government/publications/code-of-practice-... The trouble is that these set out principles, but it's hard to validate those principles without having about the same amount of knowledge as required to build an equivalent system in the first place. If you at least know the system is not connected to a WAN, you can limit the assurance required (look for WiFi funcitonality, new SSIDs, and attempts to connect to open networks), but at a certain point you need to be able to trust the vendor (else they could put a hard-coded "time bomb" into the code for the solutions they develop). I don't see much value in the academic/theoretical approaches to verification (for a consumer or stakeholder concerned by issues like these), as they tend to operate on an unrealistic set of assumptions (i.e. source code or similar levels of unrealistic access) - the reality is it could take a few days for a good embedded device hacker to even get binary firmware extracted from a device, and source code is likely a dream for products built to the lowest price overseas and imported.
Kon-Peki 2y ago
> Incidentally, why are all those panels centrally connected anyway? I’d like to know what my panels are doing, but you don’t need the internet for that. This is because of the market for carbon credits. When you installed your PV panels, someone estimated how much electricity they would generate over the next 10-15 years. Tradable carbon credits were created based on that estimate and went into the marketplace. And for the next 10-15 years they have to verify that the electricity was actually generated, or else someone has to pay back some money. Did you read the fine print on your contract? It is probably you that has to pay it back. You didn't know that one of the "rebates" you got was actually a pre-payment for those credits?!? Should have read the fine print ;) Oh yeah, BTW: that "rebate" was only your portion of the credits. The installer got some of it (and doesn't have to pay back anything), the person that filled out the paperwork you didn't know existed got some (and doesn't have to pay back anything)...
kkfx 2y ago
That's why my system (Victron + Fronius) is offline, monitored with HA, BYD battery if there is no secret in-hw backdoor in my home server can't reach the internet as well. HA can, via wireguard, to act/monitor when I'm outside my home witch might be a serious threat but it's pretty easy to cut it off if needed. There is a more important part, while with p.v. we still can go offline, with car's we can't. My car is connected and I can't do NOTHING to manage it, it's managed by it OEM behind me and that's a much bigger threat since single cars can paralyze the nation if properly blocked in critical points of the road network. At a largest scale that's the reason we can't have a national smart grid but only individual smart microgrid, meaning p.v. should be used only for self-consumption NOT grid-tied like in California.
SnorkelTan 2y ago
If solar panels can be turned off, why are utility companies having to sell excess power at a loss? Why can’t they tell the solar farms to reduce their output by the required amount?
- trebligdivad 2y ago
  In theory someone somewhere should be incentivised to spend money on building storage systems so that they then have to pay less money in the future in excess days.
- bjornsing 2y ago
  As I understand it: because the incentives are wrong. Owners of small scale solar panel installations are payed a fixed price per kWh in many EU countries, regardless of the market price. The taxpayers pick up the tab I guess.
- sanderjd 2y ago
  Solar power does get curtailed pretty often, but there isn't one uniform solution to the problem, different utilities / markets / grids have chosen different solutions to this.
- kkfx 2y ago
  It's worst actually, at least in France, if you inject to the grid you have to pay an "energy transport fee", even if you inject for free (only recently self-made systems are allowed to sell energy, before they can only donate or not inject at all) and the injected energy is now paid less than the cheapest price to the customers (6 cent/kWh for ground based p.v., 10 cent for on-roof p.v.). So well, we do not harm large utility business. What harm on scale is the variable output especially from small p.v. utilities built out of incentives NOT personal power plants, the grid is sized with some large power plants serving a large set of customers, their absorption vary but if the grid is vast (and not too vast) enough variation tend to be slow on average, let's say 50MW PP experience 100-200kW demand variation in very short time. They can compensate easily keeping the grid frequency stable. With a significant amount of grid injecting p.v. variation might be MUCH bigger creating significant stability issues where injection goes up too quickly making the frequency skyrocketing and large PP can't decrease their output fast enough risking disconnection witch in turn might put large p.v. plants offline suddenly creating a cascading effect of large blackouts. That's the real issue with grid-connected and tied renewables and another reason why we need to go toward self-consumption NOT injection.
delroth 2y ago
> In the Netherlands alone, these solar panels generate a power output equivalent to at least 25 medium sized nuclear power plants. Since this didn't pass the smell test: the author is looking at nameplate capacity, which is a completely useless metric for variable electricity production sources (a solar panel in my sunless basement has the same nameplate capacity as the same panel installed in the Sahara desert). Looking at actual yearly energy generation data, this is more like 1.5 times the generation of an average nuclear power plant (NL solar production in 2023: 21TWh, US nuclear production in 2021: 778TWh by 54 plants). Which maybe puts more into perspective the actual risks involved here. I'm not saying there shouldn't be more regulations and significantly better security practices, but otoh you could likely drive a big truck into the right power poles and cause a similar sized outage.
- epistasis 2y ago
  You are talking about energy, which is not the same thing as power. TWh == energy, GW == power. The distinction is important, especially in the Netherlands, which has a capacity factor of only about 10%-15%, whereas most of the US will be at least 20%-25%, which is twice as high. I'm not sure of the typical number of reactors in the Netherlands, but using the US average of 1.6/power plant may not be the most representative comparison.
  - delroth 2y ago
    I have no idea what you're talking about, since nowhere did I use solar capacity factor data nor did I look at number of reactors per plant.
    - epistasis 2y ago
      You are using both with your energy generated numbers. That's where they come from. Your solar TWh comes from 25GW at ~15% capacity factor, and to get your nuclear numbers you're looking at 1.6GW for each of nuclear "plants" when each reactor is usually about 1GW or less. There are ~90 reactors in the US, at 54 plants. The article is assuming 1 reactor per plant for the Netherlands.
timClicks 2y ago
Does anyone know of an inverter manufacturer that doesn't require this? Ideally, one that offers micro inverters for each panel.
- leymed 2y ago
  In greater scale, meaning power plants not the PV installed at houses, these things are taken more seriously and after purchase of equipment the control and automation of plant are in your hands. For example, Woodward, ABB have products with capacity up to 0.5 MW of single inverter. Micro inverter for each panel would be very costly. In 1 MW plant you will have around 4000 panels, communicating with that amount electronic devices would be a headache.
neilv 2y ago
> 0.002 MW - Small set of technical standards, no diplomas or certificates required Be careful with this language, especially when you're involving politicians and the non-technical. The current atrocity of criminally negligent IT infrastructure right now is mostly created and driven by people with diplomas, including from the most prestigious schools. (And a top HN story over the weekend was one of the most famous tech company execs, turned government advisor, advising students at Stanford to behave unethically, and then get enough money to pay lawyers to make the consequences go away.) And most of the certificates we do have are are individual certifications that are largely nonsense vendor training and lock-in, and these same people are then assembling and operating systems from the criminally negligent vendors. And our IT practices certifications are largely inadequate compliance theatre, to let people off the hook for actual sufficient competence. My best guess for how to start to fix this is to hold companies accountable. For example, CrowdStrike (not the worst offender, but recent example): treat it as negligence, hold them liable for all costs, which I'd guess might destroy the stock, and make C-suite and upper parts of the org chart fear prison time as a very serious investigation proceeds. I'd guess seeing that the game has changed would start to align investors and executives at other companies. What could follow next (with growing pains) is a big shakeup of the rest of the org chart and practices -- as companies figure out that they have to kill off all the culture of job-hopping, resume-driven-development, Leetcode fratbro culture, IT vendor shop fiefdoms, etc. I'd guess some companies will be wiped out as they flail around, since they'll still have too many people wired to play the old game, who will see no career option other than to try to fake it till they make it at the new, responsible game (ironically, and self-defeatingly, taking the company down with them).
- hinkley 2y ago
  Put another way: it’s far too easy and common for certification to encourage rote memorization. And only rote memorization. No higher order reasoning is imparted. Knowledge without reasoning is how you get mired in bureaucracy.
  - neilv 2y ago
    I think the larger problem is alignment. BS gatekeeping rituals and compliance-for-sale theatre are arguably just symptoms -- of companies and individuals not being aligned with developing trustworthy systems.
_trampeltier 2y ago
I don't remember when and where exactly (and didn't found it in a quick search), but there was already an incident, where an automatic update failed. I think it was something with the country code, so it was a bit isolated and not all over the world.
shermantanktop 2y ago
This article repeatedly cites the need for personnel to have diplomas, certificates, and other ceremonial bits of paper. This focus on paper qualification to mitigate risk seems a very European approach. Not saying it is wrong - it is just not emphasized as strongly elsewhere. And while it seems like a good fit for a slow-moving industry with high expectations of safety, the solar/wind world is not a slow-moving industry.
- g_p 2y ago
  A good point - perhaps the focus is too heavy on paperwork or "measurable compliance". From experience in this sector though, I think the real issue is a lack of technical awareness and competency with enough breadth to extend into the "digital" domain - often products like these are developed by people from the "power" domain (who don't necessarily recognise off the top of their head that 512-bit RSA is a #badthing and not enough to use to protect aggregated energy systems that are controllable from a single location). Clearly formal diplomas/certificates are not needed for that - some practical hands-on knowledge and experience would help a lot there. When a product gets a network interface on it, or runs programmable firmware, we should hear discussions about A/B boot, signatures, key revocation, crypto agility to enable post quantum cryptography algorithms, etc. Instead, the focus will be on low-cost development of a mobile app, controlled via the lowest-possible-cost vendor server back-end API that gets the product shipped to market quickly. Let's not even go near the "embedded system" mindset of not patching and staying up to date - embedded systems are a good place to meet Linux 2.4 or 2.6, even today... Vendors ship whatever their CPU chipset vendor gives them as a board support package, generally as a "tossed over the wall" lump of code. I doubt many of these issues (which seem to be commercial/price driven) will be resolved through paperwork, as you say.
  - shermantanktop 2y ago
    In the rest of the tech industry, what you did to get your diploma gives you about 18 months of momentum. If you haven’t learned multiple new technologies by that point, you’re in trouble. Success in this industry means perpetually redeveloping your own skills, and liking it. How someone would wave a 20 year old piece of paper as evidence that they know how to use solar tech that was developed last year, I don’t know.
WalterBright 2y ago
> It’s also possible to install new software (firmware) on the inverters via the manufacturer, either automatically or manually. As always, the vulnerability of enabling remote updates. When will people learn? Updates should only be possible if there's a physical switch (not a software switch) on the device. If it's "off", no updates are possible. Isn't the most devastating attack vector remotely installing malware? With a hardware switch, none of that malware will survive a reboot of the device. I remember when hard disk drives came with a write-enable jumper. Then, once you've made a backup, the jumper is removed. Then it is impossible to accidentally or maliciously write over your precious backup.
- DoctorOetker 2y ago
  That doesn't protect against supply chain attacks.
  - WalterBright 2y ago
    Neither does remote updating. But you'll still need physical access to the supply chain to compromise it, and that's not possible for some hacker in a basement.
    - DoctorOetker 2y ago
      I never claimed remote updating would prevent supply chain attacks. I was responding to: > With a hardware switch, none of that malware will survive a reboot of the device. A reboot of the inverter would not prevent a supply chain attack using MPPT measurement electronics for an optical backdoor channel.
      - WalterBright 2y ago
        So don't put the backdoor channel in without a physical switch.
        DoctorOetker 2y ago
veganmosfet 2y ago
Related to this topic, some research results about cybersecurity of solar inverters: https://github.com/veganmosfet/Balcony_in_the_cloud https://github.com/veganmosfet/Balcony_in_the_cloud https://github.com/veganmosfet/SolarFlareSec https://github.com/veganmosfet/SolarFlareSec https://github.com/veganmosfet/CyberEclipse https://github.com/veganmosfet/CyberEclipse https://github.com/veganmosfet/SecureWatt https://github.com/veganmosfet/SecureWatt A big mess, but it's getting slowly better...
shahzaibmushtaq 2y ago
The second figure explains a lot like everything. Cloud-based management platforms should not oversee inverters directly.
kuon 2y ago
My installer put a solaredge inverter, it took some real efforts to keep it off the cloud while injecting the data in my grafana. I can do it because I am a network engineer, but it should be easier. Anyway, I agree that there should be a regulation that forbid remote management, and you can only consult data in a read only manner remotely (you could air gap the inverter with the internet gateway using a one way rs232 connection where the inverted just write continuously). And if grid operators need to be able to turn solar off, they should install relays controlled by their infrastructure.
Derbasti 2y ago
There's a reason why I took my inverter offline after making sure that it was installed correctly. A cheap power meter now serves to measure my power generation instead.
- DoctorOetker 2y ago
  Taking it offline doesn't protect against supply chain attacks in the form of built-in kill switches. A satellite could transmit signed instructions by modulating light below the noise floor, inverters must sense the voltage/current state of the PV panels anyway for MPPT to work. Only deep inspection of the silicon and code can improve the situation. Perhaps Western blocks could develop provably secure silicon IP and code, formally verified, and perform continuous random sampling on imported goods, including full multilayer silicon inspection; publish it for free and refuse to import products that don't cooperate.
  - BenjiWiebe 2y ago
    I'm curious about the feasibility of modulating light onto a solar panel. I feel it would not be feasible, except possibly onto a single panel at a time over a long time period. Just a gut feeling based off radio stuff (GPS).
    - DoctorOetker 2y ago
      GPS can provide the coherent reference, if you mean transmitting signal (say sound) while the panel is illuminated by the sun theres youtube videos of people doing that, with a laser pointer, but in sunlight and without information theoretic justified modulation scheme. Nothing prevents the satellite to transmit the commands at night, if that feels more convincing to you. Ask yourself what is the active area of a photodiode in your TV/... ? What is the active area of your light-bucket on a roof?
      - outsomnia 2y ago
        This is just FUD.
mikewarot 2y ago
It irks me endlessly that we live in the worst timeline, where the computer equivalent of fuses and circuit breakers are almost completely unknown. Instead we trust code blindly. This results in almost all of the situations threads here address. In a better timeline, everyone has stable and secure OSs on all their devices, and the default is for everything to be locally networked, with optional monitoring from the outside via a data diode.
- DoctorOetker 2y ago
  it's incredibly hard to implement a data diode for PV systems, enemy satellites can modulate light (like a TV remote, but lower baudrate to stay below the noise floor) and an inverter could decode it and respond accordingly. They measure the PV panels anyway for MPPT.
  - Dylan16807 2y ago
    You're describing two very different concepts at the start. A data diode applies to a specific connection. It's easy to have a serial port that goes one way. Preventing any possible input to an already compromised device is much harder. But if your device isn't already compromised then it won't be looking at the input light levels for commands.
    - DoctorOetker 2y ago
      > But if your device isn't already compromised then it won't be looking at the input light levels for commands. But this is precisely part of the threat model, the manufacturers are best positioned to execute supply chain attacks on foreign buyers.
      - Dylan16807 2y ago
        That was not part of the threat model in the section of the post that mentioned data diodes. And it's vastly harder to do an attack like that.
isoprophlex 2y ago
If the west for some reason starts to vigorously argue with China over something, we're all completely fucked. They'll just tell our cheap EVs to forget how to brake, melt the firmware in our cellular towers/chips, and toggle our PV inverters off and on at a shitty time.
leymed 2y ago
Reading through comments I saw a lot of comments confusing cloud security with electrical safety of a system. Electrical protections are completely separate from communication line/ internet, has to be hard wired. As the size of plant/substation increases the automation and control system (again completely different thing from electrical protection) has its own internet system. Burning down substation, exploding transformer through solar panel is very very unrealistic. On top of that PVs installed at homes are insignificant to cause such troubles. As the size of installation increases, you will have different connection agreement and certain requirements. You can't install 15 MW and connect through inverters that are used at home, which is 100 kW at most. Even 15 MW is insignificant change for a grid.
- BlueTemplar 2y ago
  You're the one that seem to be missing something, we're not talking about local electrical safety, but about the global grid stability, with a hacker potentially hijacking software controlling tens of GW, spread out over many personal home installations of solar panels.
  - leymed 2y ago
    My point is hacking into home PV inverters doesn’t affect grid stability, you can’t penetrate into grid in that way. At worst we’re talking about losing power for a short time at those homes. When the demand is planned for a region you specifically exclude PV for load flow studies.
    - BlueTemplar 2y ago
      Why for a short time ? And this sounds reckless if most of your load flow is PV ?
      - leymed 2y ago
        Those are actually good questions. Engineering grid and PV installations acknowledge that the generation may be lost, so you are having contingency plan by means of transferring or picking the load. You're going to lose power for short time if you didn't do this properly. Actually due to the nature of PV generation, no sun no generation, it is reckless to just rely on PV. If sun is shining that is great, there'll be generation. However, daily peak consumption coincides with less day light. So during planning the target is the extreme cases (statistically estimating demand), in other words you do load flow studies for extreme cases. This helps to see your capacity limits. In parallel to this you should consider electric grid as a layered system. PV generation at house is the lowest level, so less impact. So when it is lost, or neighborhood or town lose PV generation it will impact nearby station, which is couple MW if not kW. So when you lose PV generation and you planned your system for extreme case, higher level of generation or substation will take control of it. Losing GW solar does not mean you're losing that amount power in small geographical region. You have to divide that into so many small parts. Also, PV generation at GW level is too high for small region. Hope this explanation helps. It is because how power flows, governed by rules of physics. Bottom line, if hacker wants to affect a grid they should target higher level of grid, PV panels on the rooftops will not help their cause, they are end of line.
dpedu 2y ago
I can make my computer wildly vary the amount of power it is drawing by performing different things in software. Max out the CPU and GPU load and it will instantly change from drawing ~100 watts to 500 or more. There have been plenty of botnets in the past. Some even in the millions of computers. If such a botnet decided to make every node's power draw fluctuate per above, wouldn't this cause the same type of problem? Is there a reason we've never seen this happen despite large enough networks of hacked machines existing?
- berkes 2y ago
  The Netherlands (about which the article mainly is) has 8.4 million households, let's presume they own average of one such PC you mention. A delta of 400W would mean a total consumption delta of 3.36GigaWatt. That's "peanuts" to cover. And that presumes an attacker can switch on/off all 8.4million computers in a small timeframe. 100% of them would need to be on, online and hacked. I don't think this is a realistic problem. Tesla F-ing up an OTA update that suddenly switches all charging Tesla's off, is probably a theoretical worse scenario.
  - dpedu 2y ago
    I don't doubt that that many watts is easy to cover - eventually. The problem is that it can be instantly turned on and off, whereas the grid takes time to shed load or add capacity. I found a figure on Wikipedia saying that the NL's 4.7GW worth of offshore wind capacity is 16% of their total electricity demand nationwide. 4.7/.16 = 30GW total, so this theorized computer load attack would represent about 10% of their grid's total capacity. Can their grid add and shed that much load that quickly? That's the part I doubt.
    - berkes 2y ago
      You skipped over the part where I point out that my assumptions are completely off. These numbers presume that all computers in all Dutch households are hacked, running and connected to the internet. 5% of that would be on the high side even. So a more realistic "attack" would be able to move demand, 0.5% of the total grid capacity. Switching on/off one smelter in an aluminium factory, is probably more than that. Hacking a major charging-station company and switching off their chargers is probably more than that even. I understand the direction you think, and I agree that the combined power usage of "consumer devices" is big. But the larger power system is rather well protected by an attack on these devices through the diversity of these devices and the diversity of their setup (consumer firewalls, routers, individual protection, in-house fuses, local load killswitches etc). The solar devises lacks this diversity, as the article mentions. There are few brands, and all of a brand need to connect to the one cloud service in the exact same way. So this does have a single point of attack. Whereas "switching on/off all personal computers in a country" is of an entirely different level.
ihsw 2y ago
[dead]
lysecret 2y ago
Same is true for heat pumps.
bww 2y ago
The author seems to imply, as if it were generally understood and accepted, that the reason nuclear reactors are heavily regulated is because they produce a lot of energy. Perhaps that's a component, but one really doesn't need to think about it too hard to identify better explanations for why this particular energy source is held to unusually high regulatory standards. I don't have an opinion as to whether other large-scale sources of energy should be held to similar standards, but to suggest that solar energy's failure modes are comparable to nuclear energy seems intentionally misleading.
- gwbas1c 2y ago
  Here's the critical point: > In the Netherlands alone, these solar panels generate a power output equivalent to at least 25 medium sized nuclear power plants. > Because everything runs through the manufacturer, they are able to turn all panels on and off. Or install software on the inverters so that the wrong current flows into the grid. Now, a manufacturer won’t do this intentionally, but it is easy enough to mess this up. > As an interim step, we might need to demand that control panels stick to providing pretty graphs, and make it impossible to remotely switch panels/loaders/batteries on or off. Basically, if a hacker were to make all batteries (or panels) suddenly switch between full discharge and full charge every second or so, it would tear down the electric grid. Voltage and frequency would swing rapidly, and whatever plants are riding load would struggle. This could create a massive power outage; but there is a huge risk that this could damage power plants and other infrastructure.
trollied 2y ago
There's a great Practical Engineering video about the problems solar can cause to power grids: https://www.youtube.com/watch?v=7G4ipM2qjfw https://www.youtube.com/watch?v=7G4ipM2qjfw
davedx 2y ago
Wait what. I don’t know if my inverter does what they say. For one thing the vendor went bankrupt so there is no cloud dashboard anymore. For another there are hundreds of inverter vendors not one single one. And I am highly sceptical the basic dashboard showing solar generation has some sinister inverter backdoor killswitch when the article seems to provide no evidence of such? Seriously? Edit: did some research and apparently it varies - many modern inverters can be remotely controlled by manufacturers - if they’re setup to allow it and are internet connected. The article is still sensationalist about the risks though
pshirshov 2y ago
Cloud connectivity in Victron products is optional and disabled by default. Also there is a read-only mode. The hardware is modular and the software is above any competition. Choose the responsible vendors.
kybernetyk 2y ago
No regulation? How will the private sector only do its job if we don't put it under control of incompetent bureaucrats?!
deleted 2y ago
[deleted]
amai 2y ago
Did I read Huawei in the list of solar power plant management providers?
leo69 2y ago
[dead]

456 points by ahubert 2y ago

deleted 2y ago

[deleted]

I posted this question to HN 7 months ago, more around DataCenters: >In the increasingly interconnected global economy, the reliance on Cloud Services raises questions about the national security implications of data centers. As these critical economic infrastructure sites, often strategically located underground, underwater, or in remote-cold locales, play a pivotal role, considerations arise regarding the role of military forces in safeguarding their security. While physical security measures and location obscurity provide some protection, the integration of AI into various aspects of daily life and the pervasive influence of cloud-based technologies on devices, as evident in CES GPT-enabled products, further accentuates the importance of these infrastructure sites. >Notably, instances such as the seizure of a college thesis mapping communication lines in the U.S. underscore the sensitivity of disclosing key communications infrastructure. >Companies like AWS, running data centers for the Department of Defense (DoD) and Intelligence Community (IC), demonstrate close collaboration between private entities and defense agencies. The question remains: are major cloud service providers actively involved in a national security strategy to protect the private internet infrastructure that underpins the global economy, or does the responsibility solely rest with individual companies? --- And then I posted this, based on an HNers post about mapping out Nuclear Power Plants: https://news.ycombinator.com/item?id=41189056 https://news.ycombinator.com/item?id=41189056 [We can easily map the infrastructure of the cloud and AI -- and their supply chains - and these are increasingly of National Security Concern:] ((Not to mention the actual powerplants being built to exclusively provide datacenter power)) Now, if we add the layers of the SubmarinCableMap [0] DataCenterMap [1] - and we begin to track shipments And https://i.imgur.com/zO0yz6J.png https://i.imgur.com/zO0yz6J.png -- Left is nuke, top = cables, bottom = datacenters. I went to ImportYeti to look into the NVIDIA shipments: https://i.imgur.com/k9018EC.png https://i.imgur.com/k9018EC.png And you look at the suppliers that are coming from Taiwan, such as the water-coolers and power cables to sus out where they may be shipping to, https://i.imgur.com/B5iWFQ1.png https://i.imgur.com/B5iWFQ1.png -- but instead, it would be better to find shipping lables for datacenters that are receiving containers from Taiwain, and the same suppliers as NVIDIA for things such as power cables. While the free data is out of date on ImportYeti - it gives a good supply line idea for NVIDIA... with the goal to find out which datacenters that are getting such shipments, you can begin to measure the footprint of AI as it grows, and which nuke plants they are likely powered from. Then, looking into whatever reporting one may access for the consumption/util of the nuke's capacity in various regions, we can estimate the power footprint of growing Global Compute. DataCenterNews and all sorts of datasets are available - and now the ability to create this crawler/tracker is likely full implementable https://i.imgur.com/gsM75dz.png https://i.imgur.com/gsM75dz.png https://i.imgur.com/a7nGGKh.png https://i.imgur.com/a7nGGKh.png [0] https://www.submarinecablemap.com/ https://www.submarinecablemap.com/ [1] https://www.datacentermap.com/ https://www.datacentermap.com/

dataflow 2y ago

> It’s also possible that the manufacturer gets hacked, and subsequently sends out attacker controlled and wrong software updates to the inverters, with possibly dire consequences. > There are also people that claim that the many Chinese companies managing our power panels for us might intentionally want to harm us. Who knows. Wait, seriously? The European power system relies on Chinese companies not messing it up remotely? And the debate is over whether the companies will stay nice? For heaven's sake, isn't it obvious that during a war the Chinese government can force them to just destroy the continent's power system remotely? How is this not seen as a extreme continental security risk?

formerly_proven 2y ago
Same continent that bought energy for decades from its strategic enemy. Coincidence? Probably not. Boundless naivete and corruption? Also yes.
- ragebol 2y ago
  Russia wasn't an enemy for a while. The belief was that engaging with them would ensure they wouldn't be an enemy again. That failed. Germany was an enemy once as well
amelius 2y ago
They already can by simply turning open some power mosfets in their fleet of EVs.
- bilbo0s 2y ago
  Yeah. I'm not sure everyone is really thinking clearly here. Don't get me wrong, they should get rid of this practice of cloud monitoring. A consumer should be able to access monitoring over the internet without an intermediary. They should, of course, be allowed to contract with an intermediary if that is their desire. But the security argument? Yeah, that ship has sailed. Total war, means total war. Your power grid, your internet, your communications, and your fossil fuel deliveries will all see material disruption. I wouldn't count on being able to stop those disruptions by banning a few web sites. (And frankly, during total war, those disruptions would be the least of your problems in any case.) Best bet for places like Europe, China, the US and Russia is, just don't do total war with each other. If you choose to do it anyway, then you can see what you can expect from that in the documents filed under "Play stupid games, win stupid prizes."

yetihehe 2y ago

>The owner of the panels and inverters can meanwhile establish a connection with that manufacturer using an app or website, and via the manufacturer see how their own panels are doing > It wasn’t necessary from a technical standpoint to let everything run through the manufacturer’s servers, but it was chosen to do it this way. (emphasis from article) I'm working on IoT cloud system. It was chosen to be done this way because netither consumers nor installers have any expertise whatsoever to setup their own network or any devices to be acessible from outside (and they want their panels to be accessible when they are outside their home). I can do it, most readers of HN could do it, but typical consumer or installer can't. Sad but true.

Nextgrid 2y ago
The cloud can operate as a dumb TURN relay relaying E2E-encrypted traffic. Then the worst the cloud can do is deny service to remote management (and even then, local management would still work), but it wouldn't be able to send direct control commands to the equipment since they don't have the authentication nor encryption keys. This also makes it simpler from a programming point of view - instead of having separate cloud sync & local control protocols, you just have one local protocol and you merely tunnel it through the (dumb) cloud if you can't connect directly.
- yetihehe 2y ago
  It could, but this requires to store historical data about usage on devices. If you store that encrypted data in cloud, then getting it to your mobile phone is super slow. If you store it in cloud, you can get historical data even if your device is dead or has 256 BYTES of memory and 1 megabit of flash storage. We have such devices, very effective at managing local municipal heating network and controlling several thermal controllers each via rs232 or rs485. Fortunately we preemptively moved everything into VPN'ed mobile network, we need special approval to touch anything on that network and can't connect without them granting access, so after EU started moving with cybersecurity this year, we are covered. > This also makes it simpler from a programming point of view - instead of having separate cloud sync & local control protocols, you just have one local protocol and you merely tunnel it through the (dumb) cloud if you can't connect directly. Having only cloud protocol is even simpler, I've done all of the above (I do backend and our firmwares).

dathos 2y ago

I live off-grid, power and water wise, and it really irked me that the monitoring coming with my inverter is only available online. Even when there is a network available the app will not work. I fixed this by getting a raspberry pi connected and reading it from there, but if I disconnect the inverter from the internet it will create a new network so now there is always an open network in the middle of nowhere with no option to disable it. I'm thinking about screwing it open and desoldering the wifi module but honestly I'll replace it in the next couple of years so I'd rather not kill myself by making a mistake.

ansible 2y ago
The high-voltage side should be separated from the electronics, so it shouldn't be dangerous if you are observant. It may be sufficient to just disconnect the antennas from the WiFi module, that will help prevent any network connections.
- Nextgrid 2y ago
  Disconnecting the antenna would still have leakage at close range. Grounding the antenna might be a better option. But in practice, the dangers highlighted by the article only surface when an attacker has control of many solar plants at scale. Compromising an individual one by getting close-range physical access will be a local annoyance but wouldn't scale to a level where it can threaten the grid, so it limits the pool of potential attackers to local vandals (which can achieve their goals easier by just throwing rocks at your panels).
  - ijustlovemath 2y ago
    Without an antenna, even at close range, initial handshakes will fail or be unreliable.
- serial_dev 2y ago
  Disclaimer, ymmw, if you have no clue about these systems (average people), you can still easily kill yourself in the process.

formerly_proven 2y ago

Most newer solar inverters can't even be set up without internet and most functions are only available with an always-on internet connection. This is also true for EU companies like SMA for example.

grecy 2y ago
I just installed a Fronius inverter (made in Austria) and 6.8kW of panels. The inverter itself functions perfectly fine without an internet connection, and will display instantaneous power output on the screen. I could just be content with that and look at my monthly power bill to see how much I generated and how much I used each month and never connect it to the internet. To get any kind of data logging & history from the inverter, it must be internet connected (wifi or ethernet). And all of that is through the manufacturer's website, which constantly nags me to "upgrade to pro" for some obscure feature that I'll never use.

ano-ther 2y ago

Eye opening for me. One of the arguments for renewable energy (besides emissions) has always been its potential for decentralizing power generation. Makes it more resilient, democratizes the means of production etc. This article shows that we inadvertently introduced new choke points. And of course the global security environment makes it more worrisome.

panki27 2y ago
Hmm, almost like what happend to the internet... the idea being "everything is decentralized", but now +80% of traffic passes through Cloudflare and over 90% of mails come from 2 providers!
- paxys 2y ago
  Cloudflare absolutely does not control 80% of internet traffic. I have no idea where you got that number from.
  - ezfe 2y ago
    This article says 80% of known websites, which are 19% of all websites. Probably where it came from. https://w3techs.com/technologies/details/cn-cloudflare https://w3techs.com/technologies/details/cn-cloudflare
    - paxys 2y ago
      That's still the number of websites, not their traffic. A personal blog hosted on Cloudflare and google.com are not both the same.
      - Aachen 2y ago
        Probably that just skews it worse though. Most people aren't cf customers but many of the big targets seek protection
realusername 2y ago

adolph 2y ago

The short version: most consumer and business solar panels are centrally managed by a handful of companies, mostly from countries outside of Europe. In the Netherlands alone, these solar panels generate an output equivalent to at least 25 medium sized nuclear power plants. There are almost no rules or laws in Europe governing these central administrators. . . . The same thing goes for heat pumps, home batteries, and EV charging points. Seems to me that this is very similar to the situation with IoT only with higher stakes. I appreciate this article's presentation of inverter and grid trust. Beyond trusting customer inverters to do the right thing, I wonder if there is a method for safing a grid at the hardware level. Naive question: could there be a grid provider device that prevents overcurrent or incorrectly clocked cycles?

kwhitefoot 2y ago
The utility company fuse between the property and the 240 V distribution system should prevent overcurrent. If the frequency or phase of the inverter is wrong the inverter might die first unless the network is already down. There isn't really any practical way to prevent overvoltage though. So a rogue controller in charge of all the solar systems in a street might be able to do quite a lot of damage to consumer devices. A problem from the utility point of view is that they can no longer guarantee that the 240 V side of the distribution system is safe to work on just by tripping a breaker on either side of the distribution transformer. So all work on the 240 V distribution system has to be done with the assumption that the system is live. Eventually regulations will be updated, if necessary, to deal with large numbers of solar installations on domestic buildings.
- cesarb 2y ago
  > The utility company fuse between the property and the 240 V distribution system should prevent overcurrent. If the frequency or phase of the inverter is wrong the inverter might die first unless the network is already down. To put it more simply: if the phase is wrong, the effect is the same as a short circuit, which fuses and circuit breakers protect against. If the frequency is wrong, the phase will become wrong after a number of cycles. > There isn't really any practical way to prevent overvoltage though. So a rogue controller in charge of all the solar systems in a street might be able to do quite a lot of damage to consumer devices. There is, it's called a surge protector or surge protective device (SPD). It converts any overvoltage above a certain level into a short to ground, which then trips the fuse or circuit breaker. It's often used as a protection against lightning-induced currents. > A problem from the utility point of view is that they can no longer guarantee that the 240 V side of the distribution system is safe to work on just by tripping a breaker on either side of the distribution transformer. So all work on the 240 V distribution system has to be done with the assumption that the system is live. From what I've seen, the utility workers usually ground the wiring when working on it (they have a special-purpose device for that). Once it's safely connected to ground, it's no longer live.

ThrowawayTestr 2y ago

If the general public knew how fragile the power grid is no body would be able to sleep at night.

asynchronous 2y ago
Not to be that guy, but the DOE is arguably one of the most important federal agencies in the US, and they treat the problem with the correct amount of focus, research and dedication. It’s just a very hard problem. The grid is no less secure or less resilient than it was 50 years ago, the main problem is that people are more dependent on it. Almost no one buys a personal generator before an outage happens anymore, despite it being one of the cheapest ways to get resiliency.

twoodfin 2y ago

Isn’t the right place to fix this at the junction between the plants and the grid? Regulate the grid utilities into a gateway role, and require all inverter control & telemetry traffic to pass through them. This seems likely to be more fruitful than attempting to regulate 400 Chinese panel manufacturers. What am I missing?

itishappy 2y ago
You're thinking about this right, just at a utility scale. The "plants" in this context are homes and businesses. The junction points between plants and grid are the inverters sold by the panel manufacturers.

WaitWaitWha 2y ago

Q: Are there no regulatory requirements for power plants of any kinds in EU, specially around cybersecurity? I do not allow any system into my environments (at home and at work) that requires a third party data connection function. There are way too many incidents where a provider, cloud or otherwise which required connection failed for various reasons. (e.g., Cisco Spark Board, Xerox ConnectKey, Google Cloud Print, WeWork's Connected devices, Lattice Egnines, MS Groove Music Pass, Shyp, Adobe Business Catalyst, Samsara, Zune, FuelBand, Anki Vector Robot, Google Stadia, Pebble) Despite this, I am very leery of regulating solar power specifically.

afh1 2y ago
Smartphones don't count?
- WaitWaitWha 2y ago
  Apologies, but do not understand the question. Are you suggesting using smart phones should count in "not allowing it in"? Then yes, I try to where possible. I do not depend on a smart phone. All functionality that are operationally necessary can be done elsewhere without major delays or impact.
  - afh1 2y ago
    Interesting. How do you handle MFA, do you have a special device for that? Your bank/brokerage don't require their app?
numpad0 2y ago
How would one practically verify and certify cybersecurity of a product? Even payment smartcards sometimes come with non-malicious maintenance backdoors. There seem to be little to no academic theoretical basis to this whole software security thing.
- g_p 2y ago
  Given the challenges of techniques like TLS interception (i.e. through pinning and other good security features), about the only measure I can see left is network isolation. You can set up a local network that has no WAN connectivity on it. About anything else is difficult to verify even the most basic of security properties. Certifying is another step up (although you could argue certifying is just a third party saying something passed a finite list of tests) - the real challenge is defining a meaningful certification scheme. There has been some good work towards consumer IoT device security (i.e. the 13 steps approach from the UK), that covers some of the lowest hanging fruit - https://www.gov.uk/government/publications/code-of-practice-for-consumer-iot-security/code-of-practice-for-consumer-iot-security https://www.gov.uk/government/publications/code-of-practice-... The trouble is that these set out principles, but it's hard to validate those principles without having about the same amount of knowledge as required to build an equivalent system in the first place. If you at least know the system is not connected to a WAN, you can limit the assurance required (look for WiFi funcitonality, new SSIDs, and attempts to connect to open networks), but at a certain point you need to be able to trust the vendor (else they could put a hard-coded "time bomb" into the code for the solutions they develop). I don't see much value in the academic/theoretical approaches to verification (for a consumer or stakeholder concerned by issues like these), as they tend to operate on an unrealistic set of assumptions (i.e. source code or similar levels of unrealistic access) - the reality is it could take a few days for a good embedded device hacker to even get binary firmware extracted from a device, and source code is likely a dream for products built to the lowest price overseas and imported.

Kon-Peki 2y ago

> Incidentally, why are all those panels centrally connected anyway? I’d like to know what my panels are doing, but you don’t need the internet for that. This is because of the market for carbon credits. When you installed your PV panels, someone estimated how much electricity they would generate over the next 10-15 years. Tradable carbon credits were created based on that estimate and went into the marketplace. And for the next 10-15 years they have to verify that the electricity was actually generated, or else someone has to pay back some money. Did you read the fine print on your contract? It is probably you that has to pay it back. You didn't know that one of the "rebates" you got was actually a pre-payment for those credits?!? Should have read the fine print ;) Oh yeah, BTW: that "rebate" was only your portion of the credits. The installer got some of it (and doesn't have to pay back anything), the person that filled out the paperwork you didn't know existed got some (and doesn't have to pay back anything)...

kkfx 2y ago

That's why my system (Victron + Fronius) is offline, monitored with HA, BYD battery if there is no secret in-hw backdoor in my home server can't reach the internet as well. HA can, via wireguard, to act/monitor when I'm outside my home witch might be a serious threat but it's pretty easy to cut it off if needed. There is a more important part, while with p.v. we still can go offline, with car's we can't. My car is connected and I can't do NOTHING to manage it, it's managed by it OEM behind me and that's a much bigger threat since single cars can paralyze the nation if properly blocked in critical points of the road network. At a largest scale that's the reason we can't have a national smart grid but only individual smart microgrid, meaning p.v. should be used only for self-consumption NOT grid-tied like in California.

SnorkelTan 2y ago

If solar panels can be turned off, why are utility companies having to sell excess power at a loss? Why can’t they tell the solar farms to reduce their output by the required amount?

trebligdivad 2y ago
In theory someone somewhere should be incentivised to spend money on building storage systems so that they then have to pay less money in the future in excess days.
bjornsing 2y ago
As I understand it: because the incentives are wrong. Owners of small scale solar panel installations are payed a fixed price per kWh in many EU countries, regardless of the market price. The taxpayers pick up the tab I guess.
sanderjd 2y ago
Solar power does get curtailed pretty often, but there isn't one uniform solution to the problem, different utilities / markets / grids have chosen different solutions to this.
kkfx 2y ago
It's worst actually, at least in France, if you inject to the grid you have to pay an "energy transport fee", even if you inject for free (only recently self-made systems are allowed to sell energy, before they can only donate or not inject at all) and the injected energy is now paid less than the cheapest price to the customers (6 cent/kWh for ground based p.v., 10 cent for on-roof p.v.). So well, we do not harm large utility business. What harm on scale is the variable output especially from small p.v. utilities built out of incentives NOT personal power plants, the grid is sized with some large power plants serving a large set of customers, their absorption vary but if the grid is vast (and not too vast) enough variation tend to be slow on average, let's say 50MW PP experience 100-200kW demand variation in very short time. They can compensate easily keeping the grid frequency stable. With a significant amount of grid injecting p.v. variation might be MUCH bigger creating significant stability issues where injection goes up too quickly making the frequency skyrocketing and large PP can't decrease their output fast enough risking disconnection witch in turn might put large p.v. plants offline suddenly creating a cascading effect of large blackouts. That's the real issue with grid-connected and tied renewables and another reason why we need to go toward self-consumption NOT injection.

delroth 2y ago

> In the Netherlands alone, these solar panels generate a power output equivalent to at least 25 medium sized nuclear power plants. Since this didn't pass the smell test: the author is looking at nameplate capacity, which is a completely useless metric for variable electricity production sources (a solar panel in my sunless basement has the same nameplate capacity as the same panel installed in the Sahara desert). Looking at actual yearly energy generation data, this is more like 1.5 times the generation of an average nuclear power plant (NL solar production in 2023: 21TWh, US nuclear production in 2021: 778TWh by 54 plants). Which maybe puts more into perspective the actual risks involved here. I'm not saying there shouldn't be more regulations and significantly better security practices, but otoh you could likely drive a big truck into the right power poles and cause a similar sized outage.

epistasis 2y ago
You are talking about energy, which is not the same thing as power. TWh == energy, GW == power. The distinction is important, especially in the Netherlands, which has a capacity factor of only about 10%-15%, whereas most of the US will be at least 20%-25%, which is twice as high. I'm not sure of the typical number of reactors in the Netherlands, but using the US average of 1.6/power plant may not be the most representative comparison.
- delroth 2y ago
  I have no idea what you're talking about, since nowhere did I use solar capacity factor data nor did I look at number of reactors per plant.
  - epistasis 2y ago
    You are using both with your energy generated numbers. That's where they come from. Your solar TWh comes from 25GW at ~15% capacity factor, and to get your nuclear numbers you're looking at 1.6GW for each of nuclear "plants" when each reactor is usually about 1GW or less. There are ~90 reactors in the US, at 54 plants. The article is assuming 1 reactor per plant for the Netherlands.

timClicks 2y ago

Does anyone know of an inverter manufacturer that doesn't require this? Ideally, one that offers micro inverters for each panel.

leymed 2y ago
In greater scale, meaning power plants not the PV installed at houses, these things are taken more seriously and after purchase of equipment the control and automation of plant are in your hands. For example, Woodward, ABB have products with capacity up to 0.5 MW of single inverter. Micro inverter for each panel would be very costly. In 1 MW plant you will have around 4000 panels, communicating with that amount electronic devices would be a headache.

neilv 2y ago

> 0.002 MW - Small set of technical standards, no diplomas or certificates required Be careful with this language, especially when you're involving politicians and the non-technical. The current atrocity of criminally negligent IT infrastructure right now is mostly created and driven by people with diplomas, including from the most prestigious schools. (And a top HN story over the weekend was one of the most famous tech company execs, turned government advisor, advising students at Stanford to behave unethically, and then get enough money to pay lawyers to make the consequences go away.) And most of the certificates we do have are are individual certifications that are largely nonsense vendor training and lock-in, and these same people are then assembling and operating systems from the criminally negligent vendors. And our IT practices certifications are largely inadequate compliance theatre, to let people off the hook for actual sufficient competence. My best guess for how to start to fix this is to hold companies accountable. For example, CrowdStrike (not the worst offender, but recent example): treat it as negligence, hold them liable for all costs, which I'd guess might destroy the stock, and make C-suite and upper parts of the org chart fear prison time as a very serious investigation proceeds. I'd guess seeing that the game has changed would start to align investors and executives at other companies. What could follow next (with growing pains) is a big shakeup of the rest of the org chart and practices -- as companies figure out that they have to kill off all the culture of job-hopping, resume-driven-development, Leetcode fratbro culture, IT vendor shop fiefdoms, etc. I'd guess some companies will be wiped out as they flail around, since they'll still have too many people wired to play the old game, who will see no career option other than to try to fake it till they make it at the new, responsible game (ironically, and self-defeatingly, taking the company down with them).

hinkley 2y ago
Put another way: it’s far too easy and common for certification to encourage rote memorization. And only rote memorization. No higher order reasoning is imparted. Knowledge without reasoning is how you get mired in bureaucracy.
- neilv 2y ago
  I think the larger problem is alignment. BS gatekeeping rituals and compliance-for-sale theatre are arguably just symptoms -- of companies and individuals not being aligned with developing trustworthy systems.

_trampeltier 2y ago

I don't remember when and where exactly (and didn't found it in a quick search), but there was already an incident, where an automatic update failed. I think it was something with the country code, so it was a bit isolated and not all over the world.

shermantanktop 2y ago

This article repeatedly cites the need for personnel to have diplomas, certificates, and other ceremonial bits of paper. This focus on paper qualification to mitigate risk seems a very European approach. Not saying it is wrong - it is just not emphasized as strongly elsewhere. And while it seems like a good fit for a slow-moving industry with high expectations of safety, the solar/wind world is not a slow-moving industry.

g_p 2y ago
A good point - perhaps the focus is too heavy on paperwork or "measurable compliance". From experience in this sector though, I think the real issue is a lack of technical awareness and competency with enough breadth to extend into the "digital" domain - often products like these are developed by people from the "power" domain (who don't necessarily recognise off the top of their head that 512-bit RSA is a #badthing and not enough to use to protect aggregated energy systems that are controllable from a single location). Clearly formal diplomas/certificates are not needed for that - some practical hands-on knowledge and experience would help a lot there. When a product gets a network interface on it, or runs programmable firmware, we should hear discussions about A/B boot, signatures, key revocation, crypto agility to enable post quantum cryptography algorithms, etc. Instead, the focus will be on low-cost development of a mobile app, controlled via the lowest-possible-cost vendor server back-end API that gets the product shipped to market quickly. Let's not even go near the "embedded system" mindset of not patching and staying up to date - embedded systems are a good place to meet Linux 2.4 or 2.6, even today... Vendors ship whatever their CPU chipset vendor gives them as a board support package, generally as a "tossed over the wall" lump of code. I doubt many of these issues (which seem to be commercial/price driven) will be resolved through paperwork, as you say.
- shermantanktop 2y ago
  In the rest of the tech industry, what you did to get your diploma gives you about 18 months of momentum. If you haven’t learned multiple new technologies by that point, you’re in trouble. Success in this industry means perpetually redeveloping your own skills, and liking it. How someone would wave a 20 year old piece of paper as evidence that they know how to use solar tech that was developed last year, I don’t know.

WalterBright 2y ago

> It’s also possible to install new software (firmware) on the inverters via the manufacturer, either automatically or manually. As always, the vulnerability of enabling remote updates. When will people learn? Updates should only be possible if there's a physical switch (not a software switch) on the device. If it's "off", no updates are possible. Isn't the most devastating attack vector remotely installing malware? With a hardware switch, none of that malware will survive a reboot of the device. I remember when hard disk drives came with a write-enable jumper. Then, once you've made a backup, the jumper is removed. Then it is impossible to accidentally or maliciously write over your precious backup.

DoctorOetker 2y ago
That doesn't protect against supply chain attacks.
- WalterBright 2y ago
  Neither does remote updating. But you'll still need physical access to the supply chain to compromise it, and that's not possible for some hacker in a basement.
  - DoctorOetker 2y ago
    I never claimed remote updating would prevent supply chain attacks. I was responding to: > With a hardware switch, none of that malware will survive a reboot of the device. A reboot of the inverter would not prevent a supply chain attack using MPPT measurement electronics for an optical backdoor channel.
    - WalterBright 2y ago
      So don't put the backdoor channel in without a physical switch.
      - DoctorOetker 2y ago

veganmosfet 2y ago

Related to this topic, some research results about cybersecurity of solar inverters: https://github.com/veganmosfet/Balcony_in_the_cloud https://github.com/veganmosfet/Balcony_in_the_cloud https://github.com/veganmosfet/SolarFlareSec https://github.com/veganmosfet/SolarFlareSec https://github.com/veganmosfet/CyberEclipse https://github.com/veganmosfet/CyberEclipse https://github.com/veganmosfet/SecureWatt https://github.com/veganmosfet/SecureWatt A big mess, but it's getting slowly better...

shahzaibmushtaq 2y ago

The second figure explains a lot like everything. Cloud-based management platforms should not oversee inverters directly.

kuon 2y ago

My installer put a solaredge inverter, it took some real efforts to keep it off the cloud while injecting the data in my grafana. I can do it because I am a network engineer, but it should be easier. Anyway, I agree that there should be a regulation that forbid remote management, and you can only consult data in a read only manner remotely (you could air gap the inverter with the internet gateway using a one way rs232 connection where the inverted just write continuously). And if grid operators need to be able to turn solar off, they should install relays controlled by their infrastructure.

Derbasti 2y ago

There's a reason why I took my inverter offline after making sure that it was installed correctly. A cheap power meter now serves to measure my power generation instead.

DoctorOetker 2y ago
Taking it offline doesn't protect against supply chain attacks in the form of built-in kill switches. A satellite could transmit signed instructions by modulating light below the noise floor, inverters must sense the voltage/current state of the PV panels anyway for MPPT to work. Only deep inspection of the silicon and code can improve the situation. Perhaps Western blocks could develop provably secure silicon IP and code, formally verified, and perform continuous random sampling on imported goods, including full multilayer silicon inspection; publish it for free and refuse to import products that don't cooperate.
- BenjiWiebe 2y ago
  I'm curious about the feasibility of modulating light onto a solar panel. I feel it would not be feasible, except possibly onto a single panel at a time over a long time period. Just a gut feeling based off radio stuff (GPS).
  - DoctorOetker 2y ago
    GPS can provide the coherent reference, if you mean transmitting signal (say sound) while the panel is illuminated by the sun theres youtube videos of people doing that, with a laser pointer, but in sunlight and without information theoretic justified modulation scheme. Nothing prevents the satellite to transmit the commands at night, if that feels more convincing to you. Ask yourself what is the active area of a photodiode in your TV/... ? What is the active area of your light-bucket on a roof?
    - outsomnia 2y ago
      This is just FUD.

mikewarot 2y ago

It irks me endlessly that we live in the worst timeline, where the computer equivalent of fuses and circuit breakers are almost completely unknown. Instead we trust code blindly. This results in almost all of the situations threads here address. In a better timeline, everyone has stable and secure OSs on all their devices, and the default is for everything to be locally networked, with optional monitoring from the outside via a data diode.

DoctorOetker 2y ago
it's incredibly hard to implement a data diode for PV systems, enemy satellites can modulate light (like a TV remote, but lower baudrate to stay below the noise floor) and an inverter could decode it and respond accordingly. They measure the PV panels anyway for MPPT.
- Dylan16807 2y ago
  You're describing two very different concepts at the start. A data diode applies to a specific connection. It's easy to have a serial port that goes one way. Preventing any possible input to an already compromised device is much harder. But if your device isn't already compromised then it won't be looking at the input light levels for commands.
  - DoctorOetker 2y ago
    > But if your device isn't already compromised then it won't be looking at the input light levels for commands. But this is precisely part of the threat model, the manufacturers are best positioned to execute supply chain attacks on foreign buyers.
    - Dylan16807 2y ago
      That was not part of the threat model in the section of the post that mentioned data diodes. And it's vastly harder to do an attack like that.

isoprophlex 2y ago

If the west for some reason starts to vigorously argue with China over something, we're all completely fucked. They'll just tell our cheap EVs to forget how to brake, melt the firmware in our cellular towers/chips, and toggle our PV inverters off and on at a shitty time.

leymed 2y ago

Reading through comments I saw a lot of comments confusing cloud security with electrical safety of a system. Electrical protections are completely separate from communication line/ internet, has to be hard wired. As the size of plant/substation increases the automation and control system (again completely different thing from electrical protection) has its own internet system. Burning down substation, exploding transformer through solar panel is very very unrealistic. On top of that PVs installed at homes are insignificant to cause such troubles. As the size of installation increases, you will have different connection agreement and certain requirements. You can't install 15 MW and connect through inverters that are used at home, which is 100 kW at most. Even 15 MW is insignificant change for a grid.

BlueTemplar 2y ago
You're the one that seem to be missing something, we're not talking about local electrical safety, but about the global grid stability, with a hacker potentially hijacking software controlling tens of GW, spread out over many personal home installations of solar panels.
- leymed 2y ago
  My point is hacking into home PV inverters doesn’t affect grid stability, you can’t penetrate into grid in that way. At worst we’re talking about losing power for a short time at those homes. When the demand is planned for a region you specifically exclude PV for load flow studies.
  - BlueTemplar 2y ago
    Why for a short time ? And this sounds reckless if most of your load flow is PV ?
    - leymed 2y ago
      Those are actually good questions. Engineering grid and PV installations acknowledge that the generation may be lost, so you are having contingency plan by means of transferring or picking the load. You're going to lose power for short time if you didn't do this properly. Actually due to the nature of PV generation, no sun no generation, it is reckless to just rely on PV. If sun is shining that is great, there'll be generation. However, daily peak consumption coincides with less day light. So during planning the target is the extreme cases (statistically estimating demand), in other words you do load flow studies for extreme cases. This helps to see your capacity limits. In parallel to this you should consider electric grid as a layered system. PV generation at house is the lowest level, so less impact. So when it is lost, or neighborhood or town lose PV generation it will impact nearby station, which is couple MW if not kW. So when you lose PV generation and you planned your system for extreme case, higher level of generation or substation will take control of it. Losing GW solar does not mean you're losing that amount power in small geographical region. You have to divide that into so many small parts. Also, PV generation at GW level is too high for small region. Hope this explanation helps. It is because how power flows, governed by rules of physics. Bottom line, if hacker wants to affect a grid they should target higher level of grid, PV panels on the rooftops will not help their cause, they are end of line.

dpedu 2y ago

I can make my computer wildly vary the amount of power it is drawing by performing different things in software. Max out the CPU and GPU load and it will instantly change from drawing ~100 watts to 500 or more. There have been plenty of botnets in the past. Some even in the millions of computers. If such a botnet decided to make every node's power draw fluctuate per above, wouldn't this cause the same type of problem? Is there a reason we've never seen this happen despite large enough networks of hacked machines existing?

berkes 2y ago
The Netherlands (about which the article mainly is) has 8.4 million households, let's presume they own average of one such PC you mention. A delta of 400W would mean a total consumption delta of 3.36GigaWatt. That's "peanuts" to cover. And that presumes an attacker can switch on/off all 8.4million computers in a small timeframe. 100% of them would need to be on, online and hacked. I don't think this is a realistic problem. Tesla F-ing up an OTA update that suddenly switches all charging Tesla's off, is probably a theoretical worse scenario.
- dpedu 2y ago
  I don't doubt that that many watts is easy to cover - eventually. The problem is that it can be instantly turned on and off, whereas the grid takes time to shed load or add capacity. I found a figure on Wikipedia saying that the NL's 4.7GW worth of offshore wind capacity is 16% of their total electricity demand nationwide. 4.7/.16 = 30GW total, so this theorized computer load attack would represent about 10% of their grid's total capacity. Can their grid add and shed that much load that quickly? That's the part I doubt.
  - berkes 2y ago
    You skipped over the part where I point out that my assumptions are completely off. These numbers presume that all computers in all Dutch households are hacked, running and connected to the internet. 5% of that would be on the high side even. So a more realistic "attack" would be able to move demand, 0.5% of the total grid capacity. Switching on/off one smelter in an aluminium factory, is probably more than that. Hacking a major charging-station company and switching off their chargers is probably more than that even. I understand the direction you think, and I agree that the combined power usage of "consumer devices" is big. But the larger power system is rather well protected by an attack on these devices through the diversity of these devices and the diversity of their setup (consumer firewalls, routers, individual protection, in-house fuses, local load killswitches etc). The solar devises lacks this diversity, as the article mentions. There are few brands, and all of a brand need to connect to the one cloud service in the exact same way. So this does have a single point of attack. Whereas "switching on/off all personal computers in a country" is of an entirely different level.

ihsw 2y ago

[dead]

lysecret 2y ago

Same is true for heat pumps.

bww 2y ago

The author seems to imply, as if it were generally understood and accepted, that the reason nuclear reactors are heavily regulated is because they produce a lot of energy. Perhaps that's a component, but one really doesn't need to think about it too hard to identify better explanations for why this particular energy source is held to unusually high regulatory standards. I don't have an opinion as to whether other large-scale sources of energy should be held to similar standards, but to suggest that solar energy's failure modes are comparable to nuclear energy seems intentionally misleading.

gwbas1c 2y ago
Here's the critical point: > In the Netherlands alone, these solar panels generate a power output equivalent to at least 25 medium sized nuclear power plants. > Because everything runs through the manufacturer, they are able to turn all panels on and off. Or install software on the inverters so that the wrong current flows into the grid. Now, a manufacturer won’t do this intentionally, but it is easy enough to mess this up. > As an interim step, we might need to demand that control panels stick to providing pretty graphs, and make it impossible to remotely switch panels/loaders/batteries on or off. Basically, if a hacker were to make all batteries (or panels) suddenly switch between full discharge and full charge every second or so, it would tear down the electric grid. Voltage and frequency would swing rapidly, and whatever plants are riding load would struggle. This could create a massive power outage; but there is a huge risk that this could damage power plants and other infrastructure.

trollied 2y ago

There's a great Practical Engineering video about the problems solar can cause to power grids: https://www.youtube.com/watch?v=7G4ipM2qjfw https://www.youtube.com/watch?v=7G4ipM2qjfw

davedx 2y ago

Wait what. I don’t know if my inverter does what they say. For one thing the vendor went bankrupt so there is no cloud dashboard anymore. For another there are hundreds of inverter vendors not one single one. And I am highly sceptical the basic dashboard showing solar generation has some sinister inverter backdoor killswitch when the article seems to provide no evidence of such? Seriously? Edit: did some research and apparently it varies - many modern inverters can be remotely controlled by manufacturers - if they’re setup to allow it and are internet connected. The article is still sensationalist about the risks though

pshirshov 2y ago

Cloud connectivity in Victron products is optional and disabled by default. Also there is a read-only mode. The hardware is modular and the software is above any competition. Choose the responsible vendors.

kybernetyk 2y ago

No regulation? How will the private sector only do its job if we don't put it under control of incompetent bureaucrats?!

deleted 2y ago

[deleted]

amai 2y ago

Did I read Huawei in the list of solar power plant management providers?

leo69 2y ago

[dead]