9 ms·
In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power
by oilman 1y ago
In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform.
A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.
- zdragnar 1y agoI can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
- stronglikedan 1y agoI'd rather bet on the simplicity of a glorified bicycle pump than the complexity of an engine any day, but then again, I'd probably have both!
- Symbiote 1y agoA power station can start a decently large generator with batteries. Maybe there are other concerns for an oil rig.
- tonyarkles 1y agoBatteries are great when they have charge. What happens if the generator doesn't want to start the first, second, and third time? How many start attempts do you get before the batteries are dead? The hand-pumped air compressor is the tool of last resort. You can try an engine start if there's someone there who's able to pump it. You don't have to worry about how much charge is left in your batteries or whether or not the gasoline for the 2-stroke pump engine has gone stale. It's the tool that you use as an alternative to "well, the batteries are dead too, guess we're not going to start the engine tonight... let's call the helicopters and abandon ship"
- SoftTalker 1y agoThe data center where I work has large diesel generators for power cuts. They are electric (battery) start. There is no capability to start them manually. The batteries are on maintenance chargers that keep them in good condition. The generators are started and tested every two weeks. Could the batteries be dead and the generators not start? I guess but it's very unlikely. I get that on an oil rig it might be a matter of life and death and you need some kind of manual way to bootstrap but there's not much that's more reliable than a 12V lead-acid battery and a diesel engine in good condition.
- petertodd 1y agoAlso, the data center is probably in a city, surrounded by infrastructure that could be used if necessary. An oil rig is in the middle of an ocean, and has to rely on itself.
- rainbowzootsuit 1y agoI worked with a telecom provider's data center that ended up having a quad redundant diesel generator failure during the first cold snap that took the Texas grid offline a few years back. They had at three fuel supplies gel and then failed to start. The fourth, as I remember, just didn't try to fire.
- 20after4 1y agoLead acid batteries are not exactly what I would call reliable. They require a lot of constant maintenance to ensure that they will work when you need them and they can easily degrade in such a way that they maintain voltage and appear to be good but then fail to deliver the needed amps when you demand them. This is made much worse in cold weather. Finally, if allowed to freeze when they are moderately drained, then the accumulated water inside will freeze and drastically shorten their life span. I think I'd take Lithium Ion batteries over lead acid for almost every conceivable use-case. They are superior in almost every way. Lighter, less likely to leak acid everywhere, better long term storage (due to a low self-discharge) and better cold weather discharge performance. The only drawback would be a slightly increased risk of fire with Lithium.
- Damogran6 1y agoNot being at all qualified to comment (though I work for a power company), I'd think the hand crank air compressor wouldn't suffer from no spark or bad gas.
- 0cf8612b2e1e 1y agoStale gas was my guess for the plan. Maintaining an emergency system is one of those things that is easy to neglect.
- zdragnar 1y agoIf stale gas is a concern, then all of the other steps in-between zero power and your full start are also screwed. Air compressors have more valves and gaskets that are vulnerable to oxidation, especially in salty environments, so I'd have thought the upkeep between the two, the two stroke would be easier.
- LorenPechtel 1y agoBut it's an emergency system, not a general operation system. Thus it's not going to be exposed to the salty environment most of the time. You could certainly put the whole thing in an airtight box. Look at how the military builds surface-based missiles these days: it's in a factory-sealed box. Molten salt batteries so they last for decades. (You don't see molten salt in most purposes because once it's been triggered it's lifespan is in minutes or even less. They're used in applications that only need to deliver power once.)
- lostlogin 1y agoIt’s quite funny to think about. Having good, fresh fuel on an oil rig. They need an engine that can run on crude.
- baq 1y agoDiesel will run on mostly anything if it’s running… including methane in the air intake, so you need to think quickly when presented with a generator that keeps running after cutting the fuel
- oilman 1y agoIn 99.99% of real-world scenarios, the rig would have other options to bootstrap a black start—like fully charged air tanks, backup power from a support vessel, or even emergency battery systems. The hand-cranked air compressor is really a last resort tool. We test it during commissioning to prove it could work, but in most cases, it’s never used again in the rig’s working life. It’s there for the rarest situations—like if a rig was abandoned during a hurricane, drifted off station, and someone somehow ended up back onboard without normal support. It’s a true "everything else failed" kind of backup.
- ale42 1y agoNice to see that at least in some places people are actually thinking to almost-impossible scenarios and taking them into account. I have the feeling that it's quite unlike most infrastructure development nowadays, unfortunately.
- vlachen 1y agoThe key is the responsible party's skin in that particular game. A drilling rig is a very large, very expensive, and very lucrative man-made island. The backed-up backups have backups. Not only could it be very far away from support vessels, capable of bringing it online in every situation, every minute not in production is money thrown overboard.
- ale42 1y agoVery true, although I think that economic arguments can apply to most infrastructure. What are the actual costs of a day-long nationwide blackout? I have no actual idea, but I'd not be surprised if they exceed 1 billion {EUR|USD}.
- lazide 1y agoThe part you are missing is ‘paid by whom’. Unlikely the power companies or regulator is going to be paying that amount here. It’s all the poor saps who didn’t have sufficient backup capacity. There will be costs/losses by the various power companies which weren’t generating during all this of course, but also fixing this is by definition outside of their control (the grid operators are the ones responsible). I’m sure public backlash will cause some changes of course. But the same situation in Texas didn’t result in the meaningful changes one would expect.
- morning-coffee 1y agoBased on how difficult it can be to start my chain-saw, snow-blower, and motorcycle after they've sat without being run for a while, I'd not recommend a gasoline-powered engine to be the only thing on stand-by.
- zdragnar 1y agoAir compressors in adverse environments don't hold up that well either, without basic maintenance. I've had engines run seasonally for decades. It doesn't take much for them to keep working well, though doing nothing at all is an easy way to clog up the carburator.
- ForOldHack 1y agoYou have to empty the compressor of wate. Some times a lot of water. I've seen a 33g tank with 25gs of water.
- lazide 1y agoCompressor pistons/screws that ingest grit/dirt, or aren’t run often enough to boil the water out of them, also tend to not last long. I used to help run a volunteer workshop with an Atlas Copco screw compressor, and it died in a few years because it wasn’t being run hard enough and the screws rusted (doh!).
- giantg2 1y agoIt shouldn't be that bad. A little fogging oil when put away and drain all the fuel. Then a little starting fluid on the first couple start attempts. Usually they start fairly quickly if they're in decent shape. And that's just for pull starts. My electric start mower starts right up after even 5 months of not running with stabilizer in the fuel.
- ForOldHack 1y agoGod bless Lucas stablier, and damn Joseph Lucas, prince of darkness. Any chance the stuff in Spain is made in England?
- BenjiWiebe 1y agoAs an ex small engine mechanic, I'd advise against using a 2 stroke for something like that. A 4 stroke would be a better bet. Better yet would be a natural gas/propane 4 stroke, since gasoline goes stale and plugs carburetors. Small diesels could be an option but they're harder to pull start for a given size.
- ethbr1 1y agoDo you have any small engine mechanic books you'd recommend? They're my kryptonite, but I accept it's mostly my ignorance.
- rainbowzootsuit 1y agoNot a manual in particular, but this is the Bob Ross of small engine repairs' channel: https://www.youtube.com/channel/UCr_GXW2Y56hOpGchXYNqZOQ https://www.youtube.com/channel/UCr_GXW2Y56hOpGchXYNqZOQ
- 486sx33 1y agoR. Bruce Radcliff Small Engines 4th Edition ISBN-13: 978-0826900333, ISBN-10: 082690033X
- BenjiWiebe 1y agoI learned from another mechanic, no book recommendations, sorry.
- Kon-Peki 1y ago> Small diesels could be an option but they're harder to pull start for a given size. I once needed to jump-start a small marine diesel, many miles from land... There was a small lever that cuts compression. You have to get it spinning really fast before restoring compression! It's definitely a lot of work! EDIT - Here is a cheap modern small marine diesel [1]. The operation manual suggests that you don't have to do anything to get it spinning quickly, you just have to crank it 10 times, put away the crank handle, and then flip the compression switch. That's progress! [1] https://www.yanmar.com/marine/product/engines/1gm10-marine-diesel-engine/ https://www.yanmar.com/marine/product/engines/1gm10-marine-d...
- cybergibbons25 1y agoBlack out on a rig or ship is very different to black start of a national electricity grid. Most vessels will experience a blackout periodically and the emergency generator start fine, normally on electric or stored air start, and then the main generators will come up fine. It's really not delicate, complex or tricky - some vessels have black outs happen very often, and those that don't will test it periodically. There will also be a procedure to do it manually should automation fail. There are air starters on some emergency generators that need handling pumping. These will also get tested periodically. The most complex situation during black out restoration would be manual synchronisation of generators but this is nothing compared to a black start.
- lenerdenator 1y agoI remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest. The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.
- tpmoney 1y ago> The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. Funny thing is, those are the exact words I use when talking to people about networking. And realistically anytime I dig deep into the underlying details of any big enough system I walk away with that impression. At scale, I think any system is less “controlled and planned precision” and more “harnessed chaos with a lot of resiliency to the unpredictability of that chaos”
- r3trohack3r 1y agoThis is one of the key insights in my early SRE career that changed how I viewed software engineering at scale. Components aren’t reliable. The whole thing might be duct tape and popsicle sticks. But the trick for SRE work is to create stability from unreliable components by isolating and routing around failures. It’s part of what made chaos engineering so effective. From randomly slowing down disk/network speed to unplugging server racks to making entire datacenters go dark - you intentionally introduce all sorts of crazy failure modes to intentionally break things and make sure the system remains metastable.
- ForOldHack 1y agoMessage on a mug: "if carpenters built houses the way programmers write software, a woodpecker could destroy civilization." The syncronasation of a power grid ... Wow.
- SoftTalker 1y agoHand crank? I'd think something like an oil rig would have a propane or gasoline or diesel generator with an electric start and batteries.
- detaro 1y agoand if the entire thing depends on it, you'll give that generator a handcrank as a backup too instead of assuming the batteries ever dying or getting flooded or whatever is entirely impossible.
- petertodd 1y agoThe point isn't to make a system that is easy. The point is to make a system that is guaranteed to work in any remotely realistic circumstance. In a real black start, the guys might very well grab a portable generator and just use that instead. But having the option to hand crank something rather than rely on batteries that might run flat is good.
- tantalor 1y ago> hand-crank a tiny air compressor Is that what Dr. Sattler is doing in this scene from Jurassic Park? https://www.youtube.com/watch?v=FoW4vXnkhJw https://www.youtube.com/watch?v=FoW4vXnkhJw
- optimalsolver 1y agoHold on to your culos.
- petertodd 1y agoThat would be charging up the spring to throw the breaker. High voltage breakers need to switch on (or off) very quickly, to avoid damage from arcing. It's common for them to have some kind of spring or gas piston arrangement that you pump up first to give them enough energy to do that quickly. Nice attention to detail by the filmmakers.
- TheJoeMan 1y agoNo, he's winding up a spring to close the circuit breaker quicker than a human hand could, which reduces/prevents and arc from forming as the electrical contacts close.
- Tireings 1y agoAh shit now I want that panel for my dream house
- tantalor 1y agoI found a Siemens WLL2F325 on eBay for only $24k https://www.ebay.com/itm/115854984950 https://www.ebay.com/itm/115854984950
- marai2 1y agoHow did you remember this scene?!
- ethbr1 1y agoWhen you watched Jurassic Park in a theater in your formative years, it tended to leave an impression. https://m.youtube.com/watch?v=WgQe68kF_8M https://m.youtube.com/watch?v=WgQe68kF_8M
- lillecarl 1y agoI love the "analog" handcranked air compressor to 7MW generator escalation, it really captures human ingenuity. I wonder however how being part of the "continental Europe synchronous grid" affects this, and how it isolates to Portugal and Spain like this. But yeah there are a lot of capacitors that want juice on startup that happily kills any attempt to restore power. My father had "a lot" of PA speakers at home and when we tripped the 3680w breaker (16A 220v) we had to kill some gear to get it back up again. I'm also very sure we had 230v because I lived close to the company I worked for and we ran small scale DC operations so I could monitor input voltage and frequency on SNMP so through work I had "perfect amateur" monitoring of our local grid. Just for fun I got notifications if the frequency dropped more than .1 and it happened, but rarely. Hardly ever above though since that's calibrated over time like Google handle NTP leap seconds. I love infrastructure
- WalterBright 1y agoI saw some ancient footage of an Me-109 fighter engine being started. A tech jumped on the wing and inserted a hand crank into a slot on the side. He threw all his might into turning it, and then after a delay the propeller started turning and coughed into life. I realized the tech must have been winding up a flywheel, and then the pilot engaged a clutch to dump the flywheel's inertia into the engine. The engineer in me loves the simplicity and low tech approach - a ground cart isn't needed nor is a battery charger (and batteries don't work in the cold). Perfect for a battlefield airplane. --- I saw an exhibit of an Me-262 jet fighter engine. Looking closely at the nacelle, which was cut away a bit, I noticed it enclosed a tiny piston engine. I inferred that engine was used to start the jet engine turning. It even had a pull-start handle on it! Again, no ground cart needed. --- I was reading about the MiG-15. American fighters used a pump to supply pressurized oxygen to the pilot. The MiG-15 just used a pressurized tank of air. It provided only for a limited time at altitude, but since the MiG-15 drank fuel like a drunkard, that was enough time anyway. Of course, if the ground crew forgot to pressurize it, the pilot was in trouble. Again, simple and effective.
- fsckboy 1y ago>Me-109 fighter point of trivia: Messerschmitt, yes, but Bf-109, produced by Bayerische Flugzeugwerke. you don't want to get your flugzeug works confused
- kryogen1c 1y ago> have to hand-crank a tiny air compressor just to start a small emergency generator Similarly, the US Navy maintains banks of pressurized air flasks to air-start emergency diesels. Total Capacity being some multiple of the required single-start capacity
- sgarland 1y agoOn a sub, anyway, the diesel is always started with air, not just in emergencies. Makes a cool sound as it comes up.
- iancmceachern 1y agoI understand some old radial airplane engines were started with what were essentially shotgun cartridges
- pacificmint 1y agoThey’re called Coffman engine starters [1]. Random fact: Those starters are a plot point in the 1965 film The Flight of the Phoenix, where the protagonists are trying to start a plane that’s stranded in the Sahara, but only have a small supply of starter cartridges left. [1] https://en.wikipedia.org/wiki/Coffman_engine_starter https://en.wikipedia.org/wiki/Coffman_engine_starter
- iancmceachern 1y agoLove that movie!
- _whiteCaps_ 1y agoNot just radials. The Napier Sabre H24 engine in Typhoons used cartridges as well.
- swiftcoder 1y agoI lived for a while on a sailboat equipped with an ancient Saab tractor engine (8 whole horsepower!). Was designed for cartridge starts in cold weather, though someone had fitted an electric starter by the time I saw it
- lr1970 1y agobringing islands together requires one to synchronize both -- frequency and phase. It is super difficult for large generators and transmission lines. transient heat dissipation can be a real bummer.
- ForOldHack 1y agoOne of the very most critical parts of a solar system is the grid tie.
- bill3478 1y agoThey can use GPS to synchronise or back-to-back DC if available.
- deleted 1y ago[deleted]
- Dylan16807 1y agoHow hard is getting each island within .1Hz of correct? The full grid doesn't have much trouble, but I don't know how much cutting things down impacts that. And then phase will align itself a couple times a minute so what's difficult about that part?
- chipsrafferty 1y ago[flagged]
- wepple 1y agoI can’t see an answer here: how does an air compressor start an engine?
- hollerith 1y agoThe compressor pressurizes an air tank. When the pressure in the tank is nice and high, use the compressed air to turn a turbine connected to the crankshaft of the engine.
- lr1970 1y agoThis technology of starting a diesel engine using a turbine driven by compressed air was used in Russian T-34 tank during the WWII. While Germans could not start the tanks in the cold of winter 1941 from the frozen batteries the Russians were using compressed air (hand-crank) to start T-34s just fine.
- lazide 1y agoYou can also directly feed the compressed air into a cylinder (or even the intake manifold!) to force the engine to turn. No extra turbine required, though the plumbing might get a little odd. [https://en.m.wikipedia.org/wiki/Air-start_system https://en.m.wikipedia.org/wiki/Air-start_system] That tends to be for very large engines, where the extra plumbing isn’t a problem.
- LorenPechtel 1y agoIt doesn't. It's a power storage device to allow muscle power to produce a higher peak output.
- segh 1y agoChatGPT's tone is slowly taking over the entire internet