7 ms·

I was the only person working on it for ~2 years, and I wrote the majority of the code in the first version that shipped. That said, I’m definitely glad that I

I was the only person working on it for ~2 years, and I wrote the majority of the code in the first version that shipped. That said, I’m definitely glad that I eventually found someone else (and later a whole team) to work on it with me, and it wouldn’t have been as successful without that. When people think of a binary translator, they usually just think of the ISA aspects, as opposed to the complicated interactions with the OS etc. that can consume just as much (or even more) engineering effort overall.

croemer 2y ago
That's super impressive. I remember being astonished that the x86 executable of Python running through Rosetta 2 on my M1 was just a factor of 2 slower than the native version. QEMU was something like a factor of 5-10x slower than native, IIRC.
- lostmsu 2y ago
  QEMU probably had to account for differences in memory models. A fork with that stuff removed might be able to easily catch up.
  - bonzini 2y ago
    QEMU loses a bit from being a generic translator instead of being specialized for x86->ARM like Rosetta 2, Box64 or FEXEmu. It does a lot of spilling for example even though x86 has a lot fewer registers than aarch64. Flags are also tricky, though they're pretty well optimized. In the end the main issue with them is also the spilling, but QEMU's generic architecture makes it expensive to handle consecutive jump instructions for example.
    - croemer 2y ago
      I found this blog post reverse engineering Rosetta 2 translated code: https://dougallj.wordpress.com/2022/11/09/why-is-rosetta-2-fast/ https://dougallj.wordpress.com/2022/11/09/why-is-rosetta-2-f...
      - bonzini 2y ago
        Interesting. Yeah, being able to use Arm flags always is probably a big thing, since they even added hardware support for that. It's a huge achievement for a single person to have written most of that.
        RandomThoughts3 2y ago
        > It's a huge achievement for a single person to have written most of that. Qemu was mostly Fabrice Bellard by himself at the beginning and plenty of emulators are single person project. It’s a field which lends itself well to single person development. How to properly architecture compiler/interpreter/emulator has been studied to death and everyone mostly uses the same core principles so there is little guess work as how to start (provided you have taken the time to study the field). If you are ready to do the work, you can reach a working translator from hard work alone. Then, the interesting work of optimising it starts. Don’t get me wrong, Rosetta 2 is a very impressive achievement because the performances are really good. I tip my metaphorical hat to whoever did it. My post is more in the spirit of you can do something in the same ballpark too if that’s your kick.
archagon 2y ago
It's a shame that Apple's stated intent is to throw the project away after a while. Personally, I really hope it sticks around forever, though I'm not optimistic.
- K7PJP 2y ago
  Where did Apple state that Rosetta 2 was to be deprecated?
  - larusso 2y ago
    I think they assuming from the past that this will happen. When Apple moved from powerPC to x86 there was Rosetta 1. It got deprecated as well.
    - skissane 2y ago
      I think it is different this time. A lot of developers use Rosetta 2 for Linux to run x86-64 Linux Docker containers under macOS (including me). They'll be upset if Apple discontinues Rosetta 2 for Linux. By contrast, once the PPC-to-Intel transition was under way, Rosetta was only used for running old software, and as time went by that software became increasingly outdated and use of it declined. While I think Rosetta 2 for macOS usage will likely decline over time too, I think Rosetta 2 for Linux usage is going to be much more stable and Apple will likely maintain it for a lot longer. Maybe if we eventually see a gradual migration of data centres from x86-64 to ARM, Rosetta 2 for Linux usage might begin to also decline, and then Apple may actually kill it. But, if such a migration happens, it is going to take a decade or more for us to get there.
      - CodesInChaos 2y ago
        What's the advantage of running x86-64 Linux Docker containers over running ARM Linux Docker containers? Aren't most distributionss and packages available for both platforms?
        2y ago
sitkack 2y ago
That is fascinating that this amazing system was the work of largely one person. You mentioned that interacting with the OS was super difficult. What were the most enjoyable aspects of building Rosetta?
- porphyra 2y ago
  I am also amazed that this was the work of largely one person. Having seamless and performant Rosetta 2 was a major factor why the Apple transition from Intel to Apple Silicon was viable in the first place!
zmb_ 2y ago
As someone frustrated in a team of 10+ that is struggling to ship even seemingly trivial things due to processes and overheads and inefficiencies, I would really appreciate some insights on how do you organize the work to allow a single developer to achieve this. How do you communicate with the rest of the organization? What is the lifecycle and release process like? Do you write requirements and specs for others (like validation or integration) to base their work on? Basically, what does the day to day work look like?
- tonyedgecombe 2y ago
  >How do you communicate with the rest of the organization? I wonder if Apple's renowned secrecy is a help with this. If nobody outside your small team knows what you are doing then it is hard for them to stick their oar in.
  - ladberg 2y ago
    For the record I was interning on Cameron's team while he worked on Rosetta 2 and didn't even know myself what he worked on (the rest of the team and I were working on something else). I only found out later after it was released!
    - iwontberude 2y ago
      Apple is like this, I have seen plenty of instances where you have one person carrying a team of 5 or more on their back. I always wonder how they manage to compensate them when it’s clear they are getting 10x more done. Hopefully they get paid 10x, but something tells me that isn’t true.
      - markus_zhang 2y ago
        Maybe getting interesting work is a better perk than $$, especially when Apple is already paying top dollars? I'd imagine a lot of people are willing to do things for free.
      - 2y ago

croemer 2y ago

That's super impressive. I remember being astonished that the x86 executable of Python running through Rosetta 2 on my M1 was just a factor of 2 slower than the native version. QEMU was something like a factor of 5-10x slower than native, IIRC.

lostmsu 2y ago
QEMU probably had to account for differences in memory models. A fork with that stuff removed might be able to easily catch up.
- bonzini 2y ago
  QEMU loses a bit from being a generic translator instead of being specialized for x86->ARM like Rosetta 2, Box64 or FEXEmu. It does a lot of spilling for example even though x86 has a lot fewer registers than aarch64. Flags are also tricky, though they're pretty well optimized. In the end the main issue with them is also the spilling, but QEMU's generic architecture makes it expensive to handle consecutive jump instructions for example.
  - croemer 2y ago
    I found this blog post reverse engineering Rosetta 2 translated code: https://dougallj.wordpress.com/2022/11/09/why-is-rosetta-2-fast/ https://dougallj.wordpress.com/2022/11/09/why-is-rosetta-2-f...
    - bonzini 2y ago
      Interesting. Yeah, being able to use Arm flags always is probably a big thing, since they even added hardware support for that. It's a huge achievement for a single person to have written most of that.
      - RandomThoughts3 2y ago
        > It's a huge achievement for a single person to have written most of that. Qemu was mostly Fabrice Bellard by himself at the beginning and plenty of emulators are single person project. It’s a field which lends itself well to single person development. How to properly architecture compiler/interpreter/emulator has been studied to death and everyone mostly uses the same core principles so there is little guess work as how to start (provided you have taken the time to study the field). If you are ready to do the work, you can reach a working translator from hard work alone. Then, the interesting work of optimising it starts. Don’t get me wrong, Rosetta 2 is a very impressive achievement because the performances are really good. I tip my metaphorical hat to whoever did it. My post is more in the spirit of you can do something in the same ballpark too if that’s your kick.

archagon 2y ago

It's a shame that Apple's stated intent is to throw the project away after a while. Personally, I really hope it sticks around forever, though I'm not optimistic.

K7PJP 2y ago
Where did Apple state that Rosetta 2 was to be deprecated?
- larusso 2y ago
  I think they assuming from the past that this will happen. When Apple moved from powerPC to x86 there was Rosetta 1. It got deprecated as well.
  - skissane 2y ago
    I think it is different this time. A lot of developers use Rosetta 2 for Linux to run x86-64 Linux Docker containers under macOS (including me). They'll be upset if Apple discontinues Rosetta 2 for Linux. By contrast, once the PPC-to-Intel transition was under way, Rosetta was only used for running old software, and as time went by that software became increasingly outdated and use of it declined. While I think Rosetta 2 for macOS usage will likely decline over time too, I think Rosetta 2 for Linux usage is going to be much more stable and Apple will likely maintain it for a lot longer. Maybe if we eventually see a gradual migration of data centres from x86-64 to ARM, Rosetta 2 for Linux usage might begin to also decline, and then Apple may actually kill it. But, if such a migration happens, it is going to take a decade or more for us to get there.
    - CodesInChaos 2y ago
      What's the advantage of running x86-64 Linux Docker containers over running ARM Linux Docker containers? Aren't most distributionss and packages available for both platforms?
      - 2y ago

sitkack 2y ago

That is fascinating that this amazing system was the work of largely one person. You mentioned that interacting with the OS was super difficult. What were the most enjoyable aspects of building Rosetta?

porphyra 2y ago
I am also amazed that this was the work of largely one person. Having seamless and performant Rosetta 2 was a major factor why the Apple transition from Intel to Apple Silicon was viable in the first place!

zmb_ 2y ago

As someone frustrated in a team of 10+ that is struggling to ship even seemingly trivial things due to processes and overheads and inefficiencies, I would really appreciate some insights on how do you organize the work to allow a single developer to achieve this. How do you communicate with the rest of the organization? What is the lifecycle and release process like? Do you write requirements and specs for others (like validation or integration) to base their work on? Basically, what does the day to day work look like?

tonyedgecombe 2y ago
>How do you communicate with the rest of the organization? I wonder if Apple's renowned secrecy is a help with this. If nobody outside your small team knows what you are doing then it is hard for them to stick their oar in.
- ladberg 2y ago
  For the record I was interning on Cameron's team while he worked on Rosetta 2 and didn't even know myself what he worked on (the rest of the team and I were working on something else). I only found out later after it was released!
  - iwontberude 2y ago
    Apple is like this, I have seen plenty of instances where you have one person carrying a team of 5 or more on their back. I always wonder how they manage to compensate them when it’s clear they are getting 10x more done. Hopefully they get paid 10x, but something tells me that isn’t true.
    - markus_zhang 2y ago
      Maybe getting interesting work is a better perk than $$, especially when Apple is already paying top dollars? I'd imagine a lot of people are willing to do things for free.
    - 2y ago