12 ms·

Examining circuit boards from the Space Shuttle's I/O Processor

133 points by pwg 3mo ago

kens 3mo ago
Author here if you have questions...
- joezydeco 3mo ago
  Did Shuttle fly with these bodge-wired boards or were these used for development on the ground?
  - kens 3mo ago
    I don't know if these boards were flown. They were coated with conformal coating (which I hate for reverse-engineering), which is usually omitted from prototypes. I believe that bodge wires are okay for flight if they are done properly.
    - joezydeco 3mo ago
      That's amazing. My bodge-wired work won't last 10 miles in the trunk of my car.
      - microgpt 3mo ago
        [dead]
    - HeyLaughingBoy 3mo ago
      > conformal coating (which I hate for reverse-engineering), which is usually omitted from prototypes Bad memory from a couple years ago. Debugging a machine under development at a customer site, which was luckily only a few blocks from the office, since I was there a lot, I would get random resets, hangs, and loss of debug information from the SWD probe. Swapping boards, debug probes, nothing we could think of fixed it and it was so random it was hard to track down. Long story short, after it happened on one occasion when the project EE was also debugging something and he touched one of my probes, we found that there was conformal coat residue on some of the debug pins and the probes would occasionally vibrate onto an insulated section, causing loss of signal. No idea why the board house put it on a prototype, but we probably lost a few dozen hours due to that one problem.
- ustad 3mo ago
  As always - great article. FYI - The link for Peter Kogge is broken and should probably link to https://en.wikipedia.org/wiki/Peter_Kogge https://en.wikipedia.org/wiki/Peter_Kogge
- deleted 3mo ago
  [deleted]
- inamberclad 3mo ago
  Amazing article, thanks! However, I think when you said "...heat was transmitted by convection through the metal plate inside each page...", you probably meant conduction, right? Convective heat transfer in metal would be a worrying event on the Space Shuttle!
  - kens 3mo ago
    Oops, I've fixed that now.
ck2 3mo ago
oh that is absolutely fascinating to see in detail I wonder if the very low density (relatively speaking to today) make them more robust against gamma-rays and other radiation problems once outside the atmosphere? if I remember correctly, and it's been decades of course four of the computers ran in parallel with the exact same instructions in case one failed or came up with a wrong answer and the fifth computer was the "decider" is that understanding correct? ah I see now you mention Eight networks were assigned to flight-critical systems, with each CRT display and engine controller connected to four networks for redundancy.
- pwg 3mo ago
  > very low density (relatively speaking to today) make them more robust against gamma-rays and other radiation problems once outside the atmosphere? Yes. Large size transistors (and other IC components) are less impacted by the radiation problems that exist outside the relative security of the atmosphere. Most radiation hardened IC circuity is many process sizes larger than whatever the current state of the art tiny process sizes happen to be at any given time. But note I said "less impacted". Given sufficient radiation, things will have issues, which is why items like the Shuttle carried the redundant computers, to cover for the possible lucky-strike impacts.
- kens 3mo ago
  Yes, the low density and TTL chips (instead of MOS) helped against radiation. When the Shuttle computers moved to semiconductor RAM, they needed extensive error correction, as well as a process that constantly fixed bit errors, as the memory would get multiple errors per flight due to cosmic rays. As far as redundancy, it's complicated. During critical flight phases, four computers would run the main software (PASS, Primary Avionics Software System), while the fifth computer was ready with the Backup Flight Software (BFS). The backup software was written by a completely different team to ensure that a software bug couldn't crash all the computers at once. In orbit, they used fewer redundant computers to free up computers for payload operations and stuff. The four computers constantly checked the results from each other and would vote out a faulty system. Voting ensured that a bad computer couldn't vote out the good ones (Byzantine failure). Moreover, the actuators hydraulically voted on the results from the computers: if one computer tried to push a valve in a different direction, the three good computers would physically overpower the bad computer's action at the level of the hydraulic pistons.
elzbardico 3mo ago
Glass capacitors!!!! I didn't even know this existed! and made by Corning!
- joe_mamba 3mo ago
  Corning makes some shit for the military that most civilians never even heard of.
- anitil 3mo ago
  Interestingly the earliest capacitors were glass jars (called Leyden Jars) [0]. I was taught that early inventors thought that charge was accumulated within the jar, and it wasn't until much later that it was realised that the shape was irrelevant, only the area and the distance between conductors. [0] https://en.wikipedia.org/wiki/Leyden_jar https://en.wikipedia.org/wiki/Leyden_jar
raphlinus 3mo ago
The description of the BCE reminds me a lot of the PIO in the RP2040 and 2350 microcontrollers. From the article, BCE instructions include "Transmit Data, Receive Data, Load Timeout Register, Store Status, and Wait." To me, these correspond more or less 1:1 with PIO instructions OUT, IN, SET, INT, and WAIT. These plus PUSH, PULL (which can considered auxiliaries of IN and OUT), MOV, and JMP are all the PIO instructions. Like the BCE, it runs with completely deterministic clocking, one instruction per clock, and like the BCE there are a bunch of them (a total of 12 state machines on the 2350), though they now run totally in parallel rather than being time-multiplexed. As a hobby project, I've lately been implementing USB (aiming for higher performance than Pico-PIO-USB, which proves that it's possible), and that's been quite fun. I wonder to what extent they were explicitly inspired, and to what extent you just get convergent solutions when there are similar goals and constraints.

133 points by pwg 3mo ago

kens 3mo ago

Author here if you have questions...

joezydeco 3mo ago
Did Shuttle fly with these bodge-wired boards or were these used for development on the ground?
- kens 3mo ago
  I don't know if these boards were flown. They were coated with conformal coating (which I hate for reverse-engineering), which is usually omitted from prototypes. I believe that bodge wires are okay for flight if they are done properly.
  - joezydeco 3mo ago
    That's amazing. My bodge-wired work won't last 10 miles in the trunk of my car.
    - microgpt 3mo ago
      [dead]
  - HeyLaughingBoy 3mo ago
    > conformal coating (which I hate for reverse-engineering), which is usually omitted from prototypes Bad memory from a couple years ago. Debugging a machine under development at a customer site, which was luckily only a few blocks from the office, since I was there a lot, I would get random resets, hangs, and loss of debug information from the SWD probe. Swapping boards, debug probes, nothing we could think of fixed it and it was so random it was hard to track down. Long story short, after it happened on one occasion when the project EE was also debugging something and he touched one of my probes, we found that there was conformal coat residue on some of the debug pins and the probes would occasionally vibrate onto an insulated section, causing loss of signal. No idea why the board house put it on a prototype, but we probably lost a few dozen hours due to that one problem.
ustad 3mo ago
As always - great article. FYI - The link for Peter Kogge is broken and should probably link to https://en.wikipedia.org/wiki/Peter_Kogge https://en.wikipedia.org/wiki/Peter_Kogge
deleted 3mo ago
[deleted]
inamberclad 3mo ago
Amazing article, thanks! However, I think when you said "...heat was transmitted by convection through the metal plate inside each page...", you probably meant conduction, right? Convective heat transfer in metal would be a worrying event on the Space Shuttle!
- kens 3mo ago
  Oops, I've fixed that now.

ck2 3mo ago

oh that is absolutely fascinating to see in detail I wonder if the very low density (relatively speaking to today) make them more robust against gamma-rays and other radiation problems once outside the atmosphere? if I remember correctly, and it's been decades of course four of the computers ran in parallel with the exact same instructions in case one failed or came up with a wrong answer and the fifth computer was the "decider" is that understanding correct? ah I see now you mention Eight networks were assigned to flight-critical systems, with each CRT display and engine controller connected to four networks for redundancy.

pwg 3mo ago
> very low density (relatively speaking to today) make them more robust against gamma-rays and other radiation problems once outside the atmosphere? Yes. Large size transistors (and other IC components) are less impacted by the radiation problems that exist outside the relative security of the atmosphere. Most radiation hardened IC circuity is many process sizes larger than whatever the current state of the art tiny process sizes happen to be at any given time. But note I said "less impacted". Given sufficient radiation, things will have issues, which is why items like the Shuttle carried the redundant computers, to cover for the possible lucky-strike impacts.
kens 3mo ago
Yes, the low density and TTL chips (instead of MOS) helped against radiation. When the Shuttle computers moved to semiconductor RAM, they needed extensive error correction, as well as a process that constantly fixed bit errors, as the memory would get multiple errors per flight due to cosmic rays. As far as redundancy, it's complicated. During critical flight phases, four computers would run the main software (PASS, Primary Avionics Software System), while the fifth computer was ready with the Backup Flight Software (BFS). The backup software was written by a completely different team to ensure that a software bug couldn't crash all the computers at once. In orbit, they used fewer redundant computers to free up computers for payload operations and stuff. The four computers constantly checked the results from each other and would vote out a faulty system. Voting ensured that a bad computer couldn't vote out the good ones (Byzantine failure). Moreover, the actuators hydraulically voted on the results from the computers: if one computer tried to push a valve in a different direction, the three good computers would physically overpower the bad computer's action at the level of the hydraulic pistons.

elzbardico 3mo ago

Glass capacitors!!!! I didn't even know this existed! and made by Corning!

joe_mamba 3mo ago
Corning makes some shit for the military that most civilians never even heard of.
anitil 3mo ago
Interestingly the earliest capacitors were glass jars (called Leyden Jars) [0]. I was taught that early inventors thought that charge was accumulated within the jar, and it wasn't until much later that it was realised that the shape was irrelevant, only the area and the distance between conductors. [0] https://en.wikipedia.org/wiki/Leyden_jar https://en.wikipedia.org/wiki/Leyden_jar

raphlinus 3mo ago

The description of the BCE reminds me a lot of the PIO in the RP2040 and 2350 microcontrollers. From the article, BCE instructions include "Transmit Data, Receive Data, Load Timeout Register, Store Status, and Wait." To me, these correspond more or less 1:1 with PIO instructions OUT, IN, SET, INT, and WAIT. These plus PUSH, PULL (which can considered auxiliaries of IN and OUT), MOV, and JMP are all the PIO instructions. Like the BCE, it runs with completely deterministic clocking, one instruction per clock, and like the BCE there are a bunch of them (a total of 12 state machines on the 2350), though they now run totally in parallel rather than being time-multiplexed. As a hobby project, I've lately been implementing USB (aiming for higher performance than Pico-PIO-USB, which proves that it's possible), and that's been quite fun. I wonder to what extent they were explicitly inspired, and to what extent you just get convergent solutions when there are similar goals and constraints.