10 ms·

Exploring the software that flies SpaceX rockets and starships

259 points by dinosaurs 5y ago

plq 5y ago
The article is a bit light on technical details, but the following is noteworthy: > Flight software for rockets at SpaceX is structured around the concept of a control cycle. “You read all of your inputs: sensors that we read in through an ADC, packets from the network, data from an IMU, updates from a star tracker or guidance sensor, commands from the ground,” explains Gerding. “You do some processing of those to determine your state, like where you are in the world or the status of the life support system. That determines your outputs – you write those, wait until the next tick of the clock, and then do the whole thing over again.” Wow, sounds an awful lot like a typical event loop from a game engine. Of course the main difference from a game engine would be the reliability requirements. The level of verification effort that goes into flight control software can't be comparable with the effort that goes into a game engine (assuming it's greater than zero :))
- codeulike 5y ago
  I can't really think of any other way of doing it. Interrupts? That would be bonkers though.
  - devit 5y ago
    You could abstractly structure the computation as a graph (there are many concrete approaches to that) and only recompute the parts that change due to changed input or changed intermediate results. If you have multiple outputs you can also have a scheduling/prioritization system for the subtasks. And yes, use interrupts or multiple timers to detect only changed parts without having to compare current input to previous input. It's basically the same problem as updating a browser DOM in response to application state changes.
    - tomp 5y ago
      But surely you can still do that, while maintaining compliance to the "event loop" interface...
    - codeulike 5y ago
      By 'abstractly structure the computation as a graph' you mean 'structure the journey of the rocket as a graoh'?
      - devit 5y ago
        No, as in a the graph starts from nodes corresponding to sensor inputs, with edges going to nodes that represent computations based on those, to further computations, to the outputs that drive actuators. If only one sensor input changes then intermediate computations that don't depend on it don't need to be updated.
        codeulike 5y ago
        Right. I can see how that could be useful in some situations but presumably it wouldn't work very well for a very dynamic situation like a rocket in flight (changing height, orientation, fuel, thrust every millisecond) And you still need a control loop to look for sensor changes? Its just a way of caching most of the computation?
        fighterpilot 5y ago
        It can work well for dynamic event-driven problems, with no need for a central busy loop[1]. Any sensor changes are responsible for triggering an update in any nodes that depend on it, who are responsible for triggering nodes that depend on those, and so on. One way to do it: abstract class Node, which has a list of child nodes that it loops through and alerts whenever there's been a state change in the parent node. [1] There is a busy loop but it's only there to trigger tasks on a timer.
        furiouslambda 5y ago
- sudhirj 5y ago
  Seems like a really clean and testable system, too. Can make a harness with a set of inputs, run the cycle and test outputs consistently. Performance is also easily checked, each cycle needs to run without 100ms for the 10hz systems, I guess, including garbage collection. Nice it see things kept this simple.
  - canadianfella 5y ago
    What does “without 100ms” mean?
  - WJW 5y ago
    Pretty sure they use a language without a garbage collector for the control software. Probably C or ADA. Look at the FAA specifications for more guidelines. For airplanes it's DO-178C I think, not sure about rockets. EDIT: One of the coolest projects I saw in recent time was CoPilot, which is a Haskell dialect that compiles down to C control loops which are statically guaranteed to run in constant time and memory. There is also arduino-copilot for if you want to play with ultra-hard real time software but can't afford an entire rocket.
    - helsinkiandrew 5y ago
      According to the Reddit AMA last year SpaceX use: "C & C++ for flight software, HTML, JavaScript & CSS for displays and python for testing"
  - icegreentea2 5y ago
    Yes, this stuff is nice. But it is worth noting that these control loops can often have some sort of memory, so you normally need to test over multiple cycles - you usually have a test vector to load in, and watch for the response vector. Trivial example would be if you had a PID controller implemented into your main loop. Your main loop would be storing the integral and "previous" error term.
    - bagels 5y ago
      Equivalent to handling an additionaanl sensor input and an additional output, no?
- oskhan 5y ago
  SpaceX has been hiring game developers for years now, they even showed up at GDC once to pick up game devs.
  - skizm 5y ago
    Makes sense from a financial standpoint too since game devs are known to be passionate enough to accept lower salaries if they can work on cool stuff. Not necessarily knocking it (not my preference), but it matches exactly what I hear from people that work there. The recruiters even say the pay is low, but you'll have "SpaceX" on your resume so it's worth it.
    - ilrwbwrkhv 5y ago
      Also it's world changing work. The best you can do at other faang companies is click ads.
      - noir_lord 5y ago
        I mean...Amazon has AWS which has to be fascinating to work on (though the impression I get is not a pleasant place to work).
    - valarauko 5y ago
      Do we know how much lower?
- ragebol 5y ago
  Industrial applications often use PLCs, Programmable Logic Controllers that also do this.
- holoduke 5y ago
  You could probably use some of the game "Kerbal Space Program" source code. Would be a good start to control the rockets.
  - NortySpock 5y ago
    KSP would be proprietary code designed for a simulation, not for inside a vehicle.
- chasd00 5y ago
  In all the embedded code I’ve seen there is invariably a “while(true)” somewhere. Is this not the same?
  - galangalalgol 5y ago
    Unlikely, they talk about 50hz and 10hz tasks. Real time OS often allow you to create threads that run at a given rate and must comolete. Like the 50hz task would have 20ms to complete.
    - yetihehe 5y ago
      Very likely: while(true) { ...[50hz tick tasks] if(tick%5==0) { ...[10hz tick tasks] } wait_for_next_tick(); tick++; }
      - BigMajestic 5y ago
        Nope, more likely there are internal CPU timers which emits hardware interrupts on which the tasks are performed
        chasd00 5y ago
        I guess I should also say at the bottom of those while loops there’s always some wait function attached to a timer
        Yoofie 5y ago
        No, he is correct. Alot of embedded software uses a more complex form of that exact technique that he is shown in the code. You technically can have a dedicated timer interrupt for every task but there are usually alot more tasks than HW timers, so instead they use a dedicated HW timer for time-keeping and use that as reference for all other tasks.
        yetihehe 5y ago
        Exactly. IF there is operating system used (it's not always needed), programmers implement separate OS tasks. For simpler systems (like just reading some sensors) real time os is not needed, then such loops are implemented as a template (with much more error checking and restarting). Typically that "wait for tick" function just waits until hardware timer sets overflow flag (not even full interrupt), and this is done to have simple, easy to reason about system.
- Nokinside 5y ago
  It's standard way to program in industrial automation, flight control and safety critical system. It's how digital PLC process control loop works. Programmable logic controller (PLC) https://en.wikipedia.org/wiki/Programmable_logic_controller https://en.wikipedia.org/wiki/Programmable_logic_controller
- Anarch157a 5y ago
  That's basically how an arduino works and, IIRC, the Apollo Guidance Computer.
  - xen2xen1 5y ago
    And didn't the Apollo Guidance Computer do some of it in analog?
    - jccooper 5y ago
      It had manually encoded ROM in the form of "core rope memory", which is pretty wacky, but it was a digital computer. In fact, it was the first IC computer. You can learn way too much about it and even operate (a simulation of) one here: http://www.ibiblio.org/apollo/ http://www.ibiblio.org/apollo/
- ACS_Solver 5y ago
  The article's indeed light on details, but what it describes sounds very similar to autonomous driving / driver assistance software I have experience with. That's not really surprising, as the overall system for an autonomous car and a spaceship is very similar - keep processing inputs from sensors, calculate some values and manage state machines, use those results to adjust outputs, some of which may operate actuators, and repeat. The similarity with game event loops is IMO superficial. A game typically processes events as fast as it can, and the time between any two iterations of the loop is completely unpredictable. One frame could render in 20ms and the next in 25ms, which is completely fine. For a car or spaceship though, there are hard real-time requirements. Like the article describes, some Dragon tasks run every 100ms, others run every 20ms. A lot of effort has definitely gone into making sure the tasks can complete quickly enough, with some margin, and there are definitely systems that monitor these deadlines and treat even one timing miss as a major failure.
  - hutzlibu 5y ago
    "One frame could render in 20ms and the next in 25ms, which is completely fine." No it is not fine, if you want to have a real smooth gameplay. So also in game loops, you can adjust for that, by checking timediff. But sure, the difference is that, if you miss a frame in a game loop - no real rockets crash, so there is probably (and hopefully) not as much effort put into that, like they do on SpaceX.
    - ACS_Solver 5y ago
      Framerates are averaged over much longer periods than two frames. Rendering 45 frames in 20ms each and then having a couple take 25ms just doesn't matter mostly, and addressing that is the kind of extra "nice to have" (and even then not for every kind of game). Yeah it's nice to have consistent frame times but less important than avoiding individual frames getting very slow, and a framerate drop certainly won't be treated by the engine as a critical failure that causes e.g. a crash or restart. On a hard real-time system like the Dragon, failing the deadline once is a critical error just like a game engine being unable to talk to the GPU.
    - dralley 5y ago
- detritus 5y ago
  > Wow, sounds an awful lot like a typical event loop from a game engine. I coulda sworn I recall John Carmack making a similar comparison when he was working on Armadillo Aerospace, but implying that rockets were actually a bit simpler. Apparently I misrecalled... https://twitter.com/id_aa_carmack/status/557223985977765890?lang=en https://twitter.com/id_aa_carmack/status/557223985977765890?...
- Unklejoe 5y ago
  This is also how I'd normally implement control systems in general. Sample the input, calculate, then set output.
- WrtCdEvrydy 5y ago
  This is an extremely common pattern for robotics processing... so I wonder if they just consider the rockets a robot where you're just using a finite state machine based on inputs.
- noneeeed 5y ago
  Embedded high-integrity systems have traditioanlly used this kind of cycle. It's simple to reason about and analyse. You can make assertions about worst case execution time and memory useage and you know for a fact what the order of execution is. The concurrancy on large systems like this often comes in the form of a lot of small computers talking to each other over some network/bus rather than having many systems all running on the same hardware. Building concurrant systems with predictable real-time characteristics is hard. When you have a bunch of things that really need to happen every Nth of a second in order to fly straight, a simple approach like this is definitely preferrable. In situations like this predictability tends to be more important than raw performance (assuming you can reliably hit minimums). That doesn't mean people don't use multi-threading in domains like this, but it's one of those things you want to keep as simple as you can and avoid where possible.
  - deleted 5y ago
    [deleted]
- cronix 5y ago
  > Wow, sounds an awful lot like a typical event loop from a game engine. Or Arduino's loop() method, where all the code just runs in a loop "forever."
- Teknoman117 5y ago
  The part that's way harder though is that at least in a game engine, you know the absolute world state and when you make a change to your world state, it happens exactly as you wanted. In dynamic systems, such as a rocket, robot, etc., reality is fuzzy. Your loop is mostly about "what is my best guess at my present state" and "what is my desired next state". You make changes to try and get from A to B but you may not exactly achieve B. You may have not even been in state A. The error propagates back into your next guess of state, and this repeats forever. Sensors like GPS give you an absolute position but they're imprecise. Inertial navigation is extremely accurate in telling you the change from a sample to the next, but as your position is the second integral of your acceleration, any error compounds quickly (admittedly the quality of the INS available to people willing to spend 10s of millions of dollars on a vehicle FAR exceeds what you'd be able to put in a car). Some rockets have even used ground based radar measurements uplinked to them to further improve position estimates. They probably still do this, I just don't have any hard data on current launchers.
- gwmnxnp_516a 5y ago
  The flight software is actually a real-time (aka deterministic) embedded system software. And the control loop is the typical control loop found in embedded systems: 1 - read data from sensors through ADC (Analog-To-Digital Converters), I2C, SPI, CAN (Controller Area Network) and so on; 2 - compute the output to actuators, such as motors, hydraulic cylinders, valves, motors, lights and so on, using some control law and the current state; 3 - repeat the cycle. The algorithm that drives the output may be based on algorithms from control theory, namely state space model, PID control or Kalman filter. The computers that they may be using might be single-board computers based on ARM-core, MIPS-core, PowerPC or even X86 variant for embedded systems and/or lots of microcontrollers. Before the advent of computers, the control theory algorithms were implemented using "analog computers", which are specially designed analog circuits for computing differential equations. Control theory algorithms can be developed using tools such as Matlab, Matlab-Simulink, Modelica or Scilab-Scicos (Open-Source) from Inria. Besides C or C++, another language used in embedded systems like that is Ada, which is much safer and reliable than both C and C++.
ehnto 5y ago
> This marks the beginning of a new era for SpaceX, one where it will aim to routinely fly astronauts to the ISS Does anyone have much insight into the longevity of the ISS from now? I can see it's approved for operation until 2024, so just 3 years, but could potentially continue to operate after that. If the ISS does get decommissioned, how many years does that process take, and once it's gone, what purpose does Crew Dragon serve? Not trying to be negative, hopefully by 2028 or even 2024 we will have concrete operations underway for continued space station development that could use Crew Dragon, but it does seem bold calling it a new era, when it's so precariously reliant on the ISS existing.
- nickik 5y ago
  I can answer some of that, 2024 is the current limit. But 2028 is being considered and I think its politically likely. It could probably be extend even further but it would require incensing maintenance and expensive re-qualification. NASA already has a plan however. They have a contract with Axiom Space to extend the ISS with new modules. After they have about 4 new modules they should eventually decouple after ISS and be a free floating station. NASA already has a program in planning for a commercial space station. The same way they did Commercial Crew and the moon lander. They have already asked companies to come up with plans for a station. SierraNevada really wants that contract and SpaceX will probably bid something, probably others. These would be free floating privately run stations So NASA basically hopes that sometime between 2024 and 2028 there will be two new stations and then its politically easier to drop ISS. Crew Dragon can still do free flying missions (as they will do later this year) but SpaceX hopes to replace it with Starship.
- rtkwe 5y ago
  I think the ISS will be dragged along until there's a plan for a new station which may include replacing the more ancient components instead of an entirely new station. I have a hard time seeing NASA and the US government giving up on having a continuously operated space station in LEO. What that likely means given the overall gridlock and every administration messing with the plans at NASA is that the ISS limps along until the modules themselves need to be replaced.
larrydag 5y ago
From article “We invented simple domain specific languages to express those things, such that other engineers in the company who are not software engineers can maybe configure it.” It sounds like they created their own internal API configuration scripts to be highly dynamic and configurable. Similar to many gaming or operating systems config files. Sounds be a highly productive way to test and deploy software changes. Not only for people who are not C++ proficient but also to allow Engineers and Scientist to focus on their own design work and not have to worry about software development.
ionwake 5y ago
Is it possible for a programmer with the relevant experience to apply for a role at spacex? Maybe as a consultant/contractor on non sensitive software ?
- deleted 5y ago
  [deleted]
- scanny 5y ago
  should be able to, but you do have to be a us citizen / resident I believe
- yosito 5y ago
  https://www.spacex.com/careers/index.html https://www.spacex.com/careers/index.html
- jaywalk 5y ago
  If your mention of "non sensitive software" means you're looking for a position that isn't covered by ITAR, that's not possible. Even the people working in the SpaceX cafeteria are covered by ITAR.
deleted 5y ago
[deleted]
throwaway29303 5y ago
Flight software for rockets at SpaceX is structured around the concept of a control cycle. “You read all of your inputs: sensors that we read in through an ADC, packets from the network, data from an IMU, updates from a star tracker or guidance sensor, commands from the ground,” explains Gerding. “You do some processing of those to determine your state, like where you are in the world or the status of the life support system. That determines your outputs – you write those, wait until the next tick of the clock, and then do the whole thing over again.” I didn't know what IMU meant but according to this[0] it's an Inertial Measurement Unit, I believe. [0] - https://en.wikipedia.org/wiki/Inertial_measurement_unit https://en.wikipedia.org/wiki/Inertial_measurement_unit
stunt 5y ago
If you've watched SNL, you already know that they use Tmux too.
sandGorgon 5y ago
are the 50hz chips manufactured at 50hz ? or are they downclocked to 50hz. why cant you use higher clocked speeds ? like even 500 mhz, etc ? is there something special about 10 and 50hz ?
- devit 5y ago
  That's the number of times per second the main update code runs and it has nothing to do with the number of clock cycles or instructions per second the CPU can run (other than the fact that the chip needs to be fast enough to have the update code finish before the next time it needs to start).
  - jaywalk 5y ago
    Exactly this. A chip running at 50hz would never be able to run a task 50 times per second when it's only executing 50 instructions per second.
  - sandGorgon 5y ago
    ah thanks! that makes it so much clearer.
- audunw 5y ago
  If I understood it correctly, they’re not talking about a chip literally running at 50Mhz. They’re talking about polling a sensor in a loop running on a 50Mhz timer. The processor doing that is certainly running at a much higher clock frequency
  - aero-glide2 5y ago
    To clarify, its 50Hz not 50MHz
    - abledon 5y ago
      and 50hz = 0.02s
  - 5y ago
7373737373 5y ago
The 50 Hertz rate surprises me, a lot can happen within 20ms, a lot of distance traveled at high speeds
- lukastr0 5y ago
  The question is not the distance travelled but how quickly you need to react. If you're gimballing a rocket engine, this is not going to be a device that can do much movement within 20ms.
theYipster 5y ago
Curious if the software on Dragon conforms to aviation safety critical safety assurance standards: i.e. DO178C, ARP4754
GuB-42 5y ago
I work in avionics and the principles are exactly the same. And I think it applies to all somewhat critical embedded systems. There are some details that may change, for example, we use mostly C, specialized software and a bit of ADA, no C++ in embedded systems. But the input-output periodic loops, isolation and making sure things continue working in degraded condition principles are the same. Nothing special about SpaceX here, for good reasons. In fact, some of it might be mandatory if you want to fly something that can cause serious damage if you it fails, be it a rocket or a plane.
sehugg 5y ago
FWIW, the NASA Technical Reports Server is a good resource for technical docs from Apollo to Shuttle. Many of them have enough detail to implement the algorithms. https://ntrs.nasa.gov/search?q=shuttle%20guidance https://ntrs.nasa.gov/search?q=shuttle%20guidance
polishdude20 5y ago
On the topic of rocket flight computers, here's a link to an FC I built last year for my model rocket. It does thrust vectoring and some rudimentary navigation. The control loops on this thing run at 200hz though. It's got a state machine for knowing what to do at each stage of flight as well. https://github.com/polishdude20/CygnusX1 https://github.com/polishdude20/CygnusX1
- lutorm 5y ago
  If you analyze the physics, you find that the way the inertia works out is that smaller things need shorter control cycles. The relevant timescales for large objects ends up being longer.
  - polishdude20 5y ago
    Yeah that makes sense! A lot more can go wrong on smaller things in a short amount of time
lutorm 5y ago
Note that part 2 is also out now: https://stackoverflow.blog/2021/05/11/building-a-space-based-isp/ https://stackoverflow.blog/2021/05/11/building-a-space-based...

259 points by dinosaurs 5y ago

plq 5y ago

The article is a bit light on technical details, but the following is noteworthy: > Flight software for rockets at SpaceX is structured around the concept of a control cycle. “You read all of your inputs: sensors that we read in through an ADC, packets from the network, data from an IMU, updates from a star tracker or guidance sensor, commands from the ground,” explains Gerding. “You do some processing of those to determine your state, like where you are in the world or the status of the life support system. That determines your outputs – you write those, wait until the next tick of the clock, and then do the whole thing over again.” Wow, sounds an awful lot like a typical event loop from a game engine. Of course the main difference from a game engine would be the reliability requirements. The level of verification effort that goes into flight control software can't be comparable with the effort that goes into a game engine (assuming it's greater than zero :))

codeulike 5y ago
I can't really think of any other way of doing it. Interrupts? That would be bonkers though.
- devit 5y ago
  You could abstractly structure the computation as a graph (there are many concrete approaches to that) and only recompute the parts that change due to changed input or changed intermediate results. If you have multiple outputs you can also have a scheduling/prioritization system for the subtasks. And yes, use interrupts or multiple timers to detect only changed parts without having to compare current input to previous input. It's basically the same problem as updating a browser DOM in response to application state changes.
  - tomp 5y ago
    But surely you can still do that, while maintaining compliance to the "event loop" interface...
  - codeulike 5y ago
    By 'abstractly structure the computation as a graph' you mean 'structure the journey of the rocket as a graoh'?
    - devit 5y ago
      No, as in a the graph starts from nodes corresponding to sensor inputs, with edges going to nodes that represent computations based on those, to further computations, to the outputs that drive actuators. If only one sensor input changes then intermediate computations that don't depend on it don't need to be updated.
      - codeulike 5y ago
        Right. I can see how that could be useful in some situations but presumably it wouldn't work very well for a very dynamic situation like a rocket in flight (changing height, orientation, fuel, thrust every millisecond) And you still need a control loop to look for sensor changes? Its just a way of caching most of the computation?
        fighterpilot 5y ago
        It can work well for dynamic event-driven problems, with no need for a central busy loop[1]. Any sensor changes are responsible for triggering an update in any nodes that depend on it, who are responsible for triggering nodes that depend on those, and so on. One way to do it: abstract class Node, which has a list of child nodes that it loops through and alerts whenever there's been a state change in the parent node. [1] There is a busy loop but it's only there to trigger tasks on a timer.
        furiouslambda 5y ago
sudhirj 5y ago
Seems like a really clean and testable system, too. Can make a harness with a set of inputs, run the cycle and test outputs consistently. Performance is also easily checked, each cycle needs to run without 100ms for the 10hz systems, I guess, including garbage collection. Nice it see things kept this simple.
- canadianfella 5y ago
  What does “without 100ms” mean?
- WJW 5y ago
  Pretty sure they use a language without a garbage collector for the control software. Probably C or ADA. Look at the FAA specifications for more guidelines. For airplanes it's DO-178C I think, not sure about rockets. EDIT: One of the coolest projects I saw in recent time was CoPilot, which is a Haskell dialect that compiles down to C control loops which are statically guaranteed to run in constant time and memory. There is also arduino-copilot for if you want to play with ultra-hard real time software but can't afford an entire rocket.
  - helsinkiandrew 5y ago
    According to the Reddit AMA last year SpaceX use: "C & C++ for flight software, HTML, JavaScript & CSS for displays and python for testing"
- icegreentea2 5y ago
  Yes, this stuff is nice. But it is worth noting that these control loops can often have some sort of memory, so you normally need to test over multiple cycles - you usually have a test vector to load in, and watch for the response vector. Trivial example would be if you had a PID controller implemented into your main loop. Your main loop would be storing the integral and "previous" error term.
  - bagels 5y ago
    Equivalent to handling an additionaanl sensor input and an additional output, no?
oskhan 5y ago
SpaceX has been hiring game developers for years now, they even showed up at GDC once to pick up game devs.
- skizm 5y ago
  Makes sense from a financial standpoint too since game devs are known to be passionate enough to accept lower salaries if they can work on cool stuff. Not necessarily knocking it (not my preference), but it matches exactly what I hear from people that work there. The recruiters even say the pay is low, but you'll have "SpaceX" on your resume so it's worth it.
  - ilrwbwrkhv 5y ago
    Also it's world changing work. The best you can do at other faang companies is click ads.
    - noir_lord 5y ago
      I mean...Amazon has AWS which has to be fascinating to work on (though the impression I get is not a pleasant place to work).
  - valarauko 5y ago
    Do we know how much lower?
ragebol 5y ago
Industrial applications often use PLCs, Programmable Logic Controllers that also do this.
holoduke 5y ago
You could probably use some of the game "Kerbal Space Program" source code. Would be a good start to control the rockets.
- NortySpock 5y ago
  KSP would be proprietary code designed for a simulation, not for inside a vehicle.
chasd00 5y ago
In all the embedded code I’ve seen there is invariably a “while(true)” somewhere. Is this not the same?
- galangalalgol 5y ago
  Unlikely, they talk about 50hz and 10hz tasks. Real time OS often allow you to create threads that run at a given rate and must comolete. Like the 50hz task would have 20ms to complete.
  - yetihehe 5y ago
    Very likely: while(true) { ...[50hz tick tasks] if(tick%5==0) { ...[10hz tick tasks] } wait_for_next_tick(); tick++; }
    - BigMajestic 5y ago
      Nope, more likely there are internal CPU timers which emits hardware interrupts on which the tasks are performed
      - chasd00 5y ago
        I guess I should also say at the bottom of those while loops there’s always some wait function attached to a timer
      - Yoofie 5y ago
        No, he is correct. Alot of embedded software uses a more complex form of that exact technique that he is shown in the code. You technically can have a dedicated timer interrupt for every task but there are usually alot more tasks than HW timers, so instead they use a dedicated HW timer for time-keeping and use that as reference for all other tasks.
        yetihehe 5y ago
        Exactly. IF there is operating system used (it's not always needed), programmers implement separate OS tasks. For simpler systems (like just reading some sensors) real time os is not needed, then such loops are implemented as a template (with much more error checking and restarting). Typically that "wait for tick" function just waits until hardware timer sets overflow flag (not even full interrupt), and this is done to have simple, easy to reason about system.
Nokinside 5y ago
It's standard way to program in industrial automation, flight control and safety critical system. It's how digital PLC process control loop works. Programmable logic controller (PLC) https://en.wikipedia.org/wiki/Programmable_logic_controller https://en.wikipedia.org/wiki/Programmable_logic_controller
Anarch157a 5y ago
That's basically how an arduino works and, IIRC, the Apollo Guidance Computer.
- xen2xen1 5y ago
  And didn't the Apollo Guidance Computer do some of it in analog?
  - jccooper 5y ago
    It had manually encoded ROM in the form of "core rope memory", which is pretty wacky, but it was a digital computer. In fact, it was the first IC computer. You can learn way too much about it and even operate (a simulation of) one here: http://www.ibiblio.org/apollo/ http://www.ibiblio.org/apollo/
ACS_Solver 5y ago
The article's indeed light on details, but what it describes sounds very similar to autonomous driving / driver assistance software I have experience with. That's not really surprising, as the overall system for an autonomous car and a spaceship is very similar - keep processing inputs from sensors, calculate some values and manage state machines, use those results to adjust outputs, some of which may operate actuators, and repeat. The similarity with game event loops is IMO superficial. A game typically processes events as fast as it can, and the time between any two iterations of the loop is completely unpredictable. One frame could render in 20ms and the next in 25ms, which is completely fine. For a car or spaceship though, there are hard real-time requirements. Like the article describes, some Dragon tasks run every 100ms, others run every 20ms. A lot of effort has definitely gone into making sure the tasks can complete quickly enough, with some margin, and there are definitely systems that monitor these deadlines and treat even one timing miss as a major failure.
- hutzlibu 5y ago
  "One frame could render in 20ms and the next in 25ms, which is completely fine." No it is not fine, if you want to have a real smooth gameplay. So also in game loops, you can adjust for that, by checking timediff. But sure, the difference is that, if you miss a frame in a game loop - no real rockets crash, so there is probably (and hopefully) not as much effort put into that, like they do on SpaceX.
  - ACS_Solver 5y ago
    Framerates are averaged over much longer periods than two frames. Rendering 45 frames in 20ms each and then having a couple take 25ms just doesn't matter mostly, and addressing that is the kind of extra "nice to have" (and even then not for every kind of game). Yeah it's nice to have consistent frame times but less important than avoiding individual frames getting very slow, and a framerate drop certainly won't be treated by the engine as a critical failure that causes e.g. a crash or restart. On a hard real-time system like the Dragon, failing the deadline once is a critical error just like a game engine being unable to talk to the GPU.
  - dralley 5y ago
detritus 5y ago
> Wow, sounds an awful lot like a typical event loop from a game engine. I coulda sworn I recall John Carmack making a similar comparison when he was working on Armadillo Aerospace, but implying that rockets were actually a bit simpler. Apparently I misrecalled... https://twitter.com/id_aa_carmack/status/557223985977765890?lang=en https://twitter.com/id_aa_carmack/status/557223985977765890?...
Unklejoe 5y ago
This is also how I'd normally implement control systems in general. Sample the input, calculate, then set output.
WrtCdEvrydy 5y ago
This is an extremely common pattern for robotics processing... so I wonder if they just consider the rockets a robot where you're just using a finite state machine based on inputs.
noneeeed 5y ago
Embedded high-integrity systems have traditioanlly used this kind of cycle. It's simple to reason about and analyse. You can make assertions about worst case execution time and memory useage and you know for a fact what the order of execution is. The concurrancy on large systems like this often comes in the form of a lot of small computers talking to each other over some network/bus rather than having many systems all running on the same hardware. Building concurrant systems with predictable real-time characteristics is hard. When you have a bunch of things that really need to happen every Nth of a second in order to fly straight, a simple approach like this is definitely preferrable. In situations like this predictability tends to be more important than raw performance (assuming you can reliably hit minimums). That doesn't mean people don't use multi-threading in domains like this, but it's one of those things you want to keep as simple as you can and avoid where possible.
- deleted 5y ago
  [deleted]
cronix 5y ago
> Wow, sounds an awful lot like a typical event loop from a game engine. Or Arduino's loop() method, where all the code just runs in a loop "forever."
Teknoman117 5y ago
The part that's way harder though is that at least in a game engine, you know the absolute world state and when you make a change to your world state, it happens exactly as you wanted. In dynamic systems, such as a rocket, robot, etc., reality is fuzzy. Your loop is mostly about "what is my best guess at my present state" and "what is my desired next state". You make changes to try and get from A to B but you may not exactly achieve B. You may have not even been in state A. The error propagates back into your next guess of state, and this repeats forever. Sensors like GPS give you an absolute position but they're imprecise. Inertial navigation is extremely accurate in telling you the change from a sample to the next, but as your position is the second integral of your acceleration, any error compounds quickly (admittedly the quality of the INS available to people willing to spend 10s of millions of dollars on a vehicle FAR exceeds what you'd be able to put in a car). Some rockets have even used ground based radar measurements uplinked to them to further improve position estimates. They probably still do this, I just don't have any hard data on current launchers.
gwmnxnp_516a 5y ago
The flight software is actually a real-time (aka deterministic) embedded system software. And the control loop is the typical control loop found in embedded systems: 1 - read data from sensors through ADC (Analog-To-Digital Converters), I2C, SPI, CAN (Controller Area Network) and so on; 2 - compute the output to actuators, such as motors, hydraulic cylinders, valves, motors, lights and so on, using some control law and the current state; 3 - repeat the cycle. The algorithm that drives the output may be based on algorithms from control theory, namely state space model, PID control or Kalman filter. The computers that they may be using might be single-board computers based on ARM-core, MIPS-core, PowerPC or even X86 variant for embedded systems and/or lots of microcontrollers. Before the advent of computers, the control theory algorithms were implemented using "analog computers", which are specially designed analog circuits for computing differential equations. Control theory algorithms can be developed using tools such as Matlab, Matlab-Simulink, Modelica or Scilab-Scicos (Open-Source) from Inria. Besides C or C++, another language used in embedded systems like that is Ada, which is much safer and reliable than both C and C++.

ehnto 5y ago

> This marks the beginning of a new era for SpaceX, one where it will aim to routinely fly astronauts to the ISS Does anyone have much insight into the longevity of the ISS from now? I can see it's approved for operation until 2024, so just 3 years, but could potentially continue to operate after that. If the ISS does get decommissioned, how many years does that process take, and once it's gone, what purpose does Crew Dragon serve? Not trying to be negative, hopefully by 2028 or even 2024 we will have concrete operations underway for continued space station development that could use Crew Dragon, but it does seem bold calling it a new era, when it's so precariously reliant on the ISS existing.

nickik 5y ago
I can answer some of that, 2024 is the current limit. But 2028 is being considered and I think its politically likely. It could probably be extend even further but it would require incensing maintenance and expensive re-qualification. NASA already has a plan however. They have a contract with Axiom Space to extend the ISS with new modules. After they have about 4 new modules they should eventually decouple after ISS and be a free floating station. NASA already has a program in planning for a commercial space station. The same way they did Commercial Crew and the moon lander. They have already asked companies to come up with plans for a station. SierraNevada really wants that contract and SpaceX will probably bid something, probably others. These would be free floating privately run stations So NASA basically hopes that sometime between 2024 and 2028 there will be two new stations and then its politically easier to drop ISS. Crew Dragon can still do free flying missions (as they will do later this year) but SpaceX hopes to replace it with Starship.
rtkwe 5y ago
I think the ISS will be dragged along until there's a plan for a new station which may include replacing the more ancient components instead of an entirely new station. I have a hard time seeing NASA and the US government giving up on having a continuously operated space station in LEO. What that likely means given the overall gridlock and every administration messing with the plans at NASA is that the ISS limps along until the modules themselves need to be replaced.

larrydag 5y ago

From article “We invented simple domain specific languages to express those things, such that other engineers in the company who are not software engineers can maybe configure it.” It sounds like they created their own internal API configuration scripts to be highly dynamic and configurable. Similar to many gaming or operating systems config files. Sounds be a highly productive way to test and deploy software changes. Not only for people who are not C++ proficient but also to allow Engineers and Scientist to focus on their own design work and not have to worry about software development.

ionwake 5y ago

Is it possible for a programmer with the relevant experience to apply for a role at spacex? Maybe as a consultant/contractor on non sensitive software ?

deleted 5y ago
[deleted]
scanny 5y ago
should be able to, but you do have to be a us citizen / resident I believe
yosito 5y ago
https://www.spacex.com/careers/index.html https://www.spacex.com/careers/index.html
jaywalk 5y ago
If your mention of "non sensitive software" means you're looking for a position that isn't covered by ITAR, that's not possible. Even the people working in the SpaceX cafeteria are covered by ITAR.

deleted 5y ago

[deleted]

throwaway29303 5y ago

Flight software for rockets at SpaceX is structured around the concept of a control cycle. “You read all of your inputs: sensors that we read in through an ADC, packets from the network, data from an IMU, updates from a star tracker or guidance sensor, commands from the ground,” explains Gerding. “You do some processing of those to determine your state, like where you are in the world or the status of the life support system. That determines your outputs – you write those, wait until the next tick of the clock, and then do the whole thing over again.” I didn't know what IMU meant but according to this[0] it's an Inertial Measurement Unit, I believe. [0] - https://en.wikipedia.org/wiki/Inertial_measurement_unit https://en.wikipedia.org/wiki/Inertial_measurement_unit

stunt 5y ago

If you've watched SNL, you already know that they use Tmux too.

sandGorgon 5y ago

are the 50hz chips manufactured at 50hz ? or are they downclocked to 50hz. why cant you use higher clocked speeds ? like even 500 mhz, etc ? is there something special about 10 and 50hz ?

devit 5y ago
That's the number of times per second the main update code runs and it has nothing to do with the number of clock cycles or instructions per second the CPU can run (other than the fact that the chip needs to be fast enough to have the update code finish before the next time it needs to start).
- jaywalk 5y ago
  Exactly this. A chip running at 50hz would never be able to run a task 50 times per second when it's only executing 50 instructions per second.
- sandGorgon 5y ago
  ah thanks! that makes it so much clearer.
audunw 5y ago
If I understood it correctly, they’re not talking about a chip literally running at 50Mhz. They’re talking about polling a sensor in a loop running on a 50Mhz timer. The processor doing that is certainly running at a much higher clock frequency
- aero-glide2 5y ago
  To clarify, its 50Hz not 50MHz
  - abledon 5y ago
    and 50hz = 0.02s
- 5y ago

7373737373 5y ago

The 50 Hertz rate surprises me, a lot can happen within 20ms, a lot of distance traveled at high speeds

lukastr0 5y ago
The question is not the distance travelled but how quickly you need to react. If you're gimballing a rocket engine, this is not going to be a device that can do much movement within 20ms.

theYipster 5y ago

Curious if the software on Dragon conforms to aviation safety critical safety assurance standards: i.e. DO178C, ARP4754

GuB-42 5y ago

I work in avionics and the principles are exactly the same. And I think it applies to all somewhat critical embedded systems. There are some details that may change, for example, we use mostly C, specialized software and a bit of ADA, no C++ in embedded systems. But the input-output periodic loops, isolation and making sure things continue working in degraded condition principles are the same. Nothing special about SpaceX here, for good reasons. In fact, some of it might be mandatory if you want to fly something that can cause serious damage if you it fails, be it a rocket or a plane.

sehugg 5y ago

FWIW, the NASA Technical Reports Server is a good resource for technical docs from Apollo to Shuttle. Many of them have enough detail to implement the algorithms. https://ntrs.nasa.gov/search?q=shuttle%20guidance https://ntrs.nasa.gov/search?q=shuttle%20guidance

polishdude20 5y ago

On the topic of rocket flight computers, here's a link to an FC I built last year for my model rocket. It does thrust vectoring and some rudimentary navigation. The control loops on this thing run at 200hz though. It's got a state machine for knowing what to do at each stage of flight as well. https://github.com/polishdude20/CygnusX1 https://github.com/polishdude20/CygnusX1

lutorm 5y ago
If you analyze the physics, you find that the way the inertia works out is that smaller things need shorter control cycles. The relevant timescales for large objects ends up being longer.
- polishdude20 5y ago
  Yeah that makes sense! A lot more can go wrong on smaller things in a short amount of time

lutorm 5y ago

Note that part 2 is also out now: https://stackoverflow.blog/2021/05/11/building-a-space-based-isp/ https://stackoverflow.blog/2021/05/11/building-a-space-based...