8 ms·

Huh. Finally a name for it. I do a lot of support work for Control Systems. It isn't unheard to find a chunk of PLC code that treats some sort of physical equi

Huh. Finally a name for it. I do a lot of support work for Control Systems. It isn't unheard to find a chunk of PLC code that treats some sort of physical equipment in a unique way that unintentionally creates problems. I like to parrot a line I heard elsewhere: "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born". But often enough when I find a root cause of a bug, or some sort of programmed limitation, the client wants removed. I always refuse until I can find out why that code exists. Nobody puts code in there for no reason, so I need to know why we have a timer, or an override in the first place. Often the answer is the problem it was solving no longer exists, and that's excellent, but for all the times where that code was put there to prevent something from happening and the client had a bunch of staff turnover, the original purpose is lost. Without documentation telling me why it was done that way I'm very cautious to immediately undo someone else's work. I suppose the other aspect is knowing that I trust my coworkers. They don't (typically) do something for no good reason. If it is in there, it is there for a purpose and I must trust my coworkers to have done their due diligence in the first place. If that trust breaks down then everything becomes more difficult to make decisions on.

letitbeirie 3y ago
The most haunting comment line I've ever seen was buried deep in an Allen Bradley PLC: > I don't know why this rung is needed but delete it and see what happens for yourself Did not fuck around; did not find out.
- mauvehaus 3y ago
  Context for those who haven't worked in the field: A PLC is a programmable logic controller. They are typically programmed with ladder logic which grew out of discrete relay based control systems. Generally they're controlling industrial equipment of some sort, and making changes without a thorough understanding of what's happening now and how your change will affect the equipment and process is frowned upon. https://en.wikipedia.org/wiki/Ladder_logic https://en.wikipedia.org/wiki/Ladder_logic
  - mhink 3y ago
    I interned briefly at a company which mainly built industrial control systems. One of its most interesting features (which is also very mind-bending if you're coming from any sort of typical programming ecosystem) is that every "rung" is evaluated in parallel. (As a physical relay-based control system would have back in the day.) Very wild stuff.
- sumtechguy 3y ago
  but now its like the number of licks to the center of tootsie pop the world may never know!
- praptak 3y ago
  This has the "more magic" anecdote feel to it. http://www.catb.org/jargon/html/magic-story.html http://www.catb.org/jargon/html/magic-story.html
  - w0mbat 3y ago
    I remember reading a story like this from the early days of Acorn. The first production sample of the BBC Micro came in, and would crash unexpectedly. Trial and error found that connecting a jumper wire between 2 particular points on the board stopped it crashing, but nobody could work out why it crashed or how that fixed it. They never worked it out and ended up shipping mass quantities of the BBC Micro with the magic jumper wire in place on each one.
- weaksauce 3y ago
  do you remember what the instruction was?
  - letitbeirie 3y ago
    I think it was something like |---]/[---[ONS]---[MOV]---|
woleium 3y ago
Chesterton's Fence is a principle that says change should not be made until the reasoning behind the current state of affairs is understood. It says the rash move, upon coming across a fence, would be to tear it down without understanding why it was put up.
smallpipe 3y ago
> I suppose the other aspect is knowing that I trust my coworkers. They don't (typically) do something for no good reason This so much. Depending on the git blame, I'll either remove it blindly or actually think about it way more.
Karellen 3y ago
> But often enough when I find a root cause of a bug, or some sort of programmed limitation, the client wants removed. I always refuse until I can find out why that code exists. Nobody puts code in there for no reason, so I need to know why we have a timer, or an override in the first place. Isn't that just the regular Chesterton's Fence argument though? The one the article is specifically written to point out is not enough by itself, because you need to know what else has been built with the assumption that that code is there?
- TheCapn 3y ago
  All my comment is adding a software anecdote to the story. It really is just regular Chesterton's Fence, a term I've never heard until now but dealt with for the last several years. You're not wrong, but in the context of a PLC controlling a motor or gate it is far more segregated than the code you're probably thinking of. Having a timer override on a single gate's position limit sensor would have no effect on a separate sensor/gate/motor. If the gate's function block had specific code built into it that affected all gates then what you're talking about would be more applicable.
  - bombcar 3y ago
    I think they’re thinking of things like that software control for the health machine that could get in a state where it would give a lethal dose to a patient.
deleted 3y ago
[deleted]
cortesoft 3y ago
This sounds more like the original Chesterton’s fence than what the article is describing. The article is about understanding something’s actual current purpose, rather than just the intended purpose. What the article is describing reminds me of the XKCD comic workflow: https://xkcd.com/1172/ https://xkcd.com/1172/ A system exists external to the creators original purpose, and can take on purposes that were never intended but naturally evolve. It isn’t enough to say “well that is not in the spec”, because that doesn’t change reality.
causi 3y ago
I'll never do PLC work again. Forget undocumented code, most of the time there's no schematics for the hardware you're working on because it was custom-built thirty years ago.
- TheCapn 3y ago
  My company is generally good about that. We have lots of overlapping documentation that answers questions like that in different ways. From Electrical schemas to QA docs, picture archives of panels and wiring, ticketing systems, spreadsheets over I/O, etc. etc. I hate PLC work for other reasons. I'm starting to look at going back to more traditional software role. I'm a bit tired of the road work and find the compensation for the amount asked of you to be drastically underwhelming. This meme is very much relevant: https://i.redd.it/rawo5uki1v9b1.jpg https://i.redd.it/rawo5uki1v9b1.jpg
  - causi 3y ago
    Ayep. When you're the PLC guy they expect you to know how to fix anything that plugs into a wall but isn't understood by the electricians.
bamfly 3y ago
This is why I comment a "why" for any line of code that's not incredibly obvious. And 100% of the time when it's due to interaction with something outside the codebase, whether that's an OS, filesystem, database, HTTP endpoint, hardware, whatever, if it's not some straightforward call to some API or library. Sleep due to rate limiting from another service? COMMENT. Who's requiring it, the limits if I know exactly what they are at the time (and noting that I do not and this is just an educated guess that seems to work, if not), what the system behavior might look like if it's exceeded (if I know). Using a database for something trivial the filesystem could plausibly do, but in-fact cannot for very good reasons (say, your only ergonomic way to access the FS the way that you need to, in that environment, results in resource exhaustion via runaway syscalls under load)? Comment. Workaround for a bug in some widely-used library that Ubuntu inexplicably refuses to fix in their LTS release? Comment. That kind of thing. I have written so very many "yes, I know this sucks, but here's why..." comments. I also do it when I write code that I know won't do well at larger scale, but can't be bothered to make it more scalable just then, and it doesn't need to be under current expectations (which, 99% of the time, ends up being fine indefinitely). But that may be more about protecting my ego. :-) "Yes, I know this is reading the whole file into memory, but since this is just a batch-job program with an infrequent and predictable invocation and this file is expected to be smallish... whatever. If you're running out of memory, maybe start debugging here. If you're turning this into something invoked on-demand, maybe rewrite this." At least they know I knew it'd break, LOL.
- MrBuddyCasino 3y ago
  You're doing the lords work. I often get pushback on doing this with some variation of "comments bad, code should be self-documenting". This is unwise, because there are "what code does" and "why code does" comments, but this turns out to be to nuanced to battle the meme.
  - mostlylurks 3y ago
    Self-documenting code is perfectly capable of expressing the "why" in addition to the "what". It's just that often the extra effort and/or complexity required to express the "why" through code is not worth it when a simple comment would suffice.
ke88y 3y ago
> I do a lot of support work for Control Systems. It isn't unheard to find a chunk of PLC code that treats some sort of physical equipment in a unique way that unintentionally creates problems. I like to parrot a line I heard elsewhere: "Every time Software to fix a [Electrical/Mechanical] problem, a Gremlin is born". At least some of this is cultural. EEs and MEs have historically viewed software less seriously than electrical and mechanical systems. As a result, engineering cultures dominated by EEs/MEs tend to produce shit code. Inexcusably incompetent software engineering remains common among ostensibly Professional Engineers.
- hef19898 3y ago
  On the other hand there was Juicero...
  - ke88y 3y ago
    I don't follow. Did you accidentally reply to the wrong comment?
    - hef19898 3y ago
      No, why? The point was EEs and MEs, or rather traditional engineering heavy culture produces bad software (never mind the first software devs tended to be EEs), so Juicero is good example of a software-leaning culture producing shitty hardware products.
      - overnight5349 3y ago
        Uh, the Juicero was spectacular hardware. The mechanical engineering in that beast was absolutely beautiful. I don't recall what the software was like, but none of that is why it failed, it was simply a moronic business idea. An overpriced subscription for low quality fruit in a DRM-laden pouch. Nobody wanted it then or now.
        sam_bristow 3y ago
peteradio 3y ago
> "Every time Software to fix a [Electrical/Mechanical] problem, a Gremlin is born". I think I get the gist, but that sentence is missing some words.
- TheCapn 3y ago
  oooboy is it. That's what I get for not proofreading.
hinkley 3y ago
> Nobody puts code in there for no reason, so I need to know why we have a timer, or an override in the first place. I would like to think that if I sent out an email about git hygiene that you would support me against the people who don’t understand why I get grumpy at them for commits that are fifty times as long as the commit message, and mix four concerns two of which aren’t mentioned at all. Git history is useless until you need it, and then it’s priceless. I can’t always tell what I meant by a block of code I wrote two years ago, let alone what you meant by one you wrote five years ago.
- scubbo 3y ago
  > commits that are fifty times as long as the commit message One of my proudest commits had a 1:30 commit:message length ratio. The change may have only been ~3 lines, but boy was there a lot of knowledge represented there!
  - hinkley 3y ago
    Been there, got the tshirt. The one liners can be the worst. Any time the comment has a ticket url for another product it tends to get juicy.
throw9away6 3y ago
Sometimes you have to remove the chunk and see what breaks because the person that put it in is long gone and documentation is missing
xen2xen1 3y ago
cough 737 Max cough
alex-robbins 3y ago
> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born" I'm definitely going to use this, and I think there's a more general statement: "Every time software is used to fix a problem in a lower layer (which may also be software), a gremlin is born."
glonq 3y ago
> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born" Early in my career, I was confused by seemingly-crazy questions in the Hacker Test (https://www-users.york.ac.uk/~ss44/joke/hacker.htm https://www-users.york.ac.uk/~ss44/joke/hacker.htm) like... > 0133 Ever fix a hardware problem in software? > 0134 ... Vice versa? But after spending years developing embedded systems, I don't even blink at such questions. Yes, of course I have committed such necessarily evils!
HeyLaughingBoy 3y ago
> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born". Just did one of those this morning. Hmmm.
Izkata 3y ago
> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born" Gizmo caca. (...I just watched both Gremlins movies last weekend...)
dacox 3y ago
The unspoken thing here is that PLC code often(usually?) isn't exactly written in text, or in a format readable by anything other than the PLC programming software. After a year long foray into the world of PLC, I felt like I was programming in the dark ages. I'm assuming its a bit better at very big plants/operations, but still.
kvmet 3y ago
I work with control systems and have a similar mantra: "You can't overcome physics with software." It's super common to have someone ask if a mechanical/human/electrical/process issue can be fixed with software because people believe that programming time is free. Sometimes it's not even that it's impossible to do in software, but adding unnecessary complexity almost always backfires and you'll wind up fixing it the right way anyway in the end.
deleted 3y ago
[deleted]

letitbeirie 3y ago

The most haunting comment line I've ever seen was buried deep in an Allen Bradley PLC: > I don't know why this rung is needed but delete it and see what happens for yourself Did not fuck around; did not find out.

mauvehaus 3y ago
Context for those who haven't worked in the field: A PLC is a programmable logic controller. They are typically programmed with ladder logic which grew out of discrete relay based control systems. Generally they're controlling industrial equipment of some sort, and making changes without a thorough understanding of what's happening now and how your change will affect the equipment and process is frowned upon. https://en.wikipedia.org/wiki/Ladder_logic https://en.wikipedia.org/wiki/Ladder_logic
- mhink 3y ago
  I interned briefly at a company which mainly built industrial control systems. One of its most interesting features (which is also very mind-bending if you're coming from any sort of typical programming ecosystem) is that every "rung" is evaluated in parallel. (As a physical relay-based control system would have back in the day.) Very wild stuff.
sumtechguy 3y ago
but now its like the number of licks to the center of tootsie pop the world may never know!
praptak 3y ago
This has the "more magic" anecdote feel to it. http://www.catb.org/jargon/html/magic-story.html http://www.catb.org/jargon/html/magic-story.html
- w0mbat 3y ago
  I remember reading a story like this from the early days of Acorn. The first production sample of the BBC Micro came in, and would crash unexpectedly. Trial and error found that connecting a jumper wire between 2 particular points on the board stopped it crashing, but nobody could work out why it crashed or how that fixed it. They never worked it out and ended up shipping mass quantities of the BBC Micro with the magic jumper wire in place on each one.
weaksauce 3y ago
do you remember what the instruction was?
- letitbeirie 3y ago
  I think it was something like |---]/[---[ONS]---[MOV]---|

woleium 3y ago

Chesterton's Fence is a principle that says change should not be made until the reasoning behind the current state of affairs is understood. It says the rash move, upon coming across a fence, would be to tear it down without understanding why it was put up.

smallpipe 3y ago

> I suppose the other aspect is knowing that I trust my coworkers. They don't (typically) do something for no good reason This so much. Depending on the git blame, I'll either remove it blindly or actually think about it way more.

Karellen 3y ago

> But often enough when I find a root cause of a bug, or some sort of programmed limitation, the client wants removed. I always refuse until I can find out why that code exists. Nobody puts code in there for no reason, so I need to know why we have a timer, or an override in the first place. Isn't that just the regular Chesterton's Fence argument though? The one the article is specifically written to point out is not enough by itself, because you need to know what else has been built with the assumption that that code is there?

TheCapn 3y ago
All my comment is adding a software anecdote to the story. It really is just regular Chesterton's Fence, a term I've never heard until now but dealt with for the last several years. You're not wrong, but in the context of a PLC controlling a motor or gate it is far more segregated than the code you're probably thinking of. Having a timer override on a single gate's position limit sensor would have no effect on a separate sensor/gate/motor. If the gate's function block had specific code built into it that affected all gates then what you're talking about would be more applicable.
- bombcar 3y ago
  I think they’re thinking of things like that software control for the health machine that could get in a state where it would give a lethal dose to a patient.

deleted 3y ago

[deleted]

cortesoft 3y ago

This sounds more like the original Chesterton’s fence than what the article is describing. The article is about understanding something’s actual current purpose, rather than just the intended purpose. What the article is describing reminds me of the XKCD comic workflow: https://xkcd.com/1172/ https://xkcd.com/1172/ A system exists external to the creators original purpose, and can take on purposes that were never intended but naturally evolve. It isn’t enough to say “well that is not in the spec”, because that doesn’t change reality.

causi 3y ago

I'll never do PLC work again. Forget undocumented code, most of the time there's no schematics for the hardware you're working on because it was custom-built thirty years ago.

TheCapn 3y ago
My company is generally good about that. We have lots of overlapping documentation that answers questions like that in different ways. From Electrical schemas to QA docs, picture archives of panels and wiring, ticketing systems, spreadsheets over I/O, etc. etc. I hate PLC work for other reasons. I'm starting to look at going back to more traditional software role. I'm a bit tired of the road work and find the compensation for the amount asked of you to be drastically underwhelming. This meme is very much relevant: https://i.redd.it/rawo5uki1v9b1.jpg https://i.redd.it/rawo5uki1v9b1.jpg
- causi 3y ago
  Ayep. When you're the PLC guy they expect you to know how to fix anything that plugs into a wall but isn't understood by the electricians.

bamfly 3y ago

This is why I comment a "why" for any line of code that's not incredibly obvious. And 100% of the time when it's due to interaction with something outside the codebase, whether that's an OS, filesystem, database, HTTP endpoint, hardware, whatever, if it's not some straightforward call to some API or library. Sleep due to rate limiting from another service? COMMENT. Who's requiring it, the limits if I know exactly what they are at the time (and noting that I do not and this is just an educated guess that seems to work, if not), what the system behavior might look like if it's exceeded (if I know). Using a database for something trivial the filesystem could plausibly do, but in-fact cannot for very good reasons (say, your only ergonomic way to access the FS the way that you need to, in that environment, results in resource exhaustion via runaway syscalls under load)? Comment. Workaround for a bug in some widely-used library that Ubuntu inexplicably refuses to fix in their LTS release? Comment. That kind of thing. I have written so very many "yes, I know this sucks, but here's why..." comments. I also do it when I write code that I know won't do well at larger scale, but can't be bothered to make it more scalable just then, and it doesn't need to be under current expectations (which, 99% of the time, ends up being fine indefinitely). But that may be more about protecting my ego. :-) "Yes, I know this is reading the whole file into memory, but since this is just a batch-job program with an infrequent and predictable invocation and this file is expected to be smallish... whatever. If you're running out of memory, maybe start debugging here. If you're turning this into something invoked on-demand, maybe rewrite this." At least they know I knew it'd break, LOL.

MrBuddyCasino 3y ago
You're doing the lords work. I often get pushback on doing this with some variation of "comments bad, code should be self-documenting". This is unwise, because there are "what code does" and "why code does" comments, but this turns out to be to nuanced to battle the meme.
- mostlylurks 3y ago
  Self-documenting code is perfectly capable of expressing the "why" in addition to the "what". It's just that often the extra effort and/or complexity required to express the "why" through code is not worth it when a simple comment would suffice.

ke88y 3y ago

> I do a lot of support work for Control Systems. It isn't unheard to find a chunk of PLC code that treats some sort of physical equipment in a unique way that unintentionally creates problems. I like to parrot a line I heard elsewhere: "Every time Software to fix a [Electrical/Mechanical] problem, a Gremlin is born". At least some of this is cultural. EEs and MEs have historically viewed software less seriously than electrical and mechanical systems. As a result, engineering cultures dominated by EEs/MEs tend to produce shit code. Inexcusably incompetent software engineering remains common among ostensibly Professional Engineers.

hef19898 3y ago
On the other hand there was Juicero...
- ke88y 3y ago
  I don't follow. Did you accidentally reply to the wrong comment?
  - hef19898 3y ago
    No, why? The point was EEs and MEs, or rather traditional engineering heavy culture produces bad software (never mind the first software devs tended to be EEs), so Juicero is good example of a software-leaning culture producing shitty hardware products.
    - overnight5349 3y ago
      Uh, the Juicero was spectacular hardware. The mechanical engineering in that beast was absolutely beautiful. I don't recall what the software was like, but none of that is why it failed, it was simply a moronic business idea. An overpriced subscription for low quality fruit in a DRM-laden pouch. Nobody wanted it then or now.
      - sam_bristow 3y ago

peteradio 3y ago

> "Every time Software to fix a [Electrical/Mechanical] problem, a Gremlin is born". I think I get the gist, but that sentence is missing some words.

TheCapn 3y ago
oooboy is it. That's what I get for not proofreading.

hinkley 3y ago

> Nobody puts code in there for no reason, so I need to know why we have a timer, or an override in the first place. I would like to think that if I sent out an email about git hygiene that you would support me against the people who don’t understand why I get grumpy at them for commits that are fifty times as long as the commit message, and mix four concerns two of which aren’t mentioned at all. Git history is useless until you need it, and then it’s priceless. I can’t always tell what I meant by a block of code I wrote two years ago, let alone what you meant by one you wrote five years ago.

scubbo 3y ago
> commits that are fifty times as long as the commit message One of my proudest commits had a 1:30 commit:message length ratio. The change may have only been ~3 lines, but boy was there a lot of knowledge represented there!
- hinkley 3y ago
  Been there, got the tshirt. The one liners can be the worst. Any time the comment has a ticket url for another product it tends to get juicy.

throw9away6 3y ago

Sometimes you have to remove the chunk and see what breaks because the person that put it in is long gone and documentation is missing

xen2xen1 3y ago

cough 737 Max cough

alex-robbins 3y ago

> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born" I'm definitely going to use this, and I think there's a more general statement: "Every time software is used to fix a problem in a lower layer (which may also be software), a gremlin is born."

glonq 3y ago

> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born" Early in my career, I was confused by seemingly-crazy questions in the Hacker Test (https://www-users.york.ac.uk/~ss44/joke/hacker.htm https://www-users.york.ac.uk/~ss44/joke/hacker.htm) like... > 0133 Ever fix a hardware problem in software? > 0134 ... Vice versa? But after spending years developing embedded systems, I don't even blink at such questions. Yes, of course I have committed such necessarily evils!

HeyLaughingBoy 3y ago

> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born". Just did one of those this morning. Hmmm.

Izkata 3y ago

> "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born" Gizmo caca. (...I just watched both Gremlins movies last weekend...)

dacox 3y ago

The unspoken thing here is that PLC code often(usually?) isn't exactly written in text, or in a format readable by anything other than the PLC programming software. After a year long foray into the world of PLC, I felt like I was programming in the dark ages. I'm assuming its a bit better at very big plants/operations, but still.

kvmet 3y ago

I work with control systems and have a similar mantra: "You can't overcome physics with software." It's super common to have someone ask if a mechanical/human/electrical/process issue can be fixed with software because people believe that programming time is free. Sometimes it's not even that it's impossible to do in software, but adding unnecessary complexity almost always backfires and you'll wind up fixing it the right way anyway in the end.

deleted 3y ago

[deleted]