8 ms·

Designing an Ethernet Switch ASIC

252 points by random__duck 2mo ago

noncoml 2mo ago
“ Cut-through vs Store-and-Forward” I had a Google interview collapse because the last interviewer was a Principle Engineer who just couldn’t accept that a switch will ever forward a frame before reading in full and checking the FCS. I was too much of an idiot to go along with it and instead ended up having an argument with him.
- random__duck 2mo ago
  Sounds like they could have used you on that team. Bad luck, but it at least you can walk away proud knowing it was not an issue with your technical skills. :)
- inigyou 2mo ago
  Cut-through is applicable in a minority of scenarios which are sometimes common. The majority of switches, especially the ones that support any advanced features at all, can't use it. Even when you can use it, you still can't use it all the time because of port conflicts. As far as advanced features go, even adding and removing VLAN tags is a headache in a cut-through switch.
  - userbinator 2mo ago
    As far as advanced features go, even adding and removing VLAN tags is a headache in a cut-through switch. Not really; it just needs to inject the tag right after the source address, or not pass it through at all; and of course recompute throughout and replace the FCS at the end. Other more advanced packet editing can be done in a streaming fashion too, with the latency only limited by the length of any "forward references" needed.
    - crote 2mo ago
      But "just recompute the FCS" isn't quite the right approach, is it? With a traditional cut-through switch the entire packet is passed as-is, so you're not modifying a corrupt package. You don't drop it, but the corruption is trivially detectable on the other side. If you naively recompute the FCS on transmit you essentially mark a corrupted package as valid, so you have to compute the FCS on transmit and receive, and intentionally send out the wrong FCS on transmit if you determine the receive FCS is invalid.
      - bcrl 2mo ago
        It's easy enough to write a bit of Verilog to perform partial updates on the CRC to only compensate for what has changed. I did so for an FPGA project I implemented a number of years ago when processing VLAN tags and doing IP forwarding. That said, there are lazy hardware designers that just slap on a block to recompute the CRC and replace the old one with the new. An experience I had years ago at Red Hat taught me that the hard way as a shiny new Cisco ethernet switch IT deployed ended up costing us weeks of frustration and delayed a release when it caused bit flips in NFS packets during stress testing. The switch was so awesomely advanced that it recomputed the CRC on the bit flipped packets causing kernel builds to fail. <sigh>
        foobiekr 2mo ago
        Wow that rings a bell. The lack of end to end ECC protection in some packet paths on very high end ASICs is something I just can’t get over.
        bcrl 2mo ago
        ECC tends to be too expensive to be used widely in hardware data paths. Most of the time hardware will use a parity bit as checking it is far less costly (both in terms of gates and power). Parity bits are usually sufficient, as the purpose is really limited to identifying hardware failure. The CRC should prevent delivery of a corrupted packet provided the hardware doesn't cheat and blindly recompute it.
- sophacles 2mo ago
  I'm struggling to think of any real-world networking scenario where cut-through switching for ethernet can provide any sort of benefit. Most data-center networks are involve aggregation/spine/core switches at higher speed than TOR/leaf switches - so the packet has to be buffered. Most every workload is going to involve port collisions (2 packets at the same time needing the same output port) - again a full buffering required (or drops and throughput reduction). Bigger networks start using vlans, mpls, ip routing at switches, and vxlan - requiring more lookups per-packet, larger "minimum chunks of packet seen before port is determined" sizes, etc reducing the value of cut-through. And so on. But i also struggle to find a scenario where a principle engineer at google is refusing to accept that cut-through switches exist, but rather, was refusing to accept that the scenario they're talking through with you would ever involve an effectively deployed cut-through switch (since that's basically never).
  - noncoml 2mo ago
    > but rather, was refusing to accept that the scenario they're talking through with you would ever involve an effectively deployed cut-through switch (since that's basically never). Nope, the argument was exactly if a cut-through switch exists or ever existed Edit: Actually the conversation came back to me: He asked me about the frame format. Once I drew that on the board, he asked why is the DST before the SRC. My answer was in order for the switch to start doing the lookups ASAP and that's when I mentioned that a cut-through switch can switch the packet before it even finishes receiving it. After that we spend the rest of the interview time arguing if cut-through switches actually exist or not. Both he and I having too big of an ego to let it go..
    - icedchai 2mo ago
      So he didn't want to google it?
      - stinkbeetle 2mo ago
        I see you've never had a run-in with a blowhard in a position of power. Lucky.
- gjvc 2mo ago
  "principal"
- Aurornis 2mo ago
  I remember studying cut-through forwarding in a class and thinking it was obviously going to be used everywhere. Then I was disappointed to enter the real world of networking and learn that it wasn’t actually used as often as I had been led to believe due to conflicting with higher priority features. So the interview may have been a conflict between textbook learning and the Google engineer’s practical experience. It’s tough to encounter that conflict in interviews because you get two people talking past each other.
inigyou 2mo ago
Nice! I've been following waferspace and tiny tapeout loosely but I'm sadly not in a position to be able to work on something right now. I'm surprised you managed to fit something like this in a TT grid square.
- random__duck 2mo ago
  You can join multiple Tiny Tapeout tiles (grid squares) together to form bigger projects, which is what I did here. It's actually one of the bigger 4 grid projects. Also how much you can fit really depends on the node the shuttle is targeting. So I was in luck that this was on Global Foundries 180nm as you can cram in a lot for flip-flops onto a single tile compared with Skywater 130nm. On the other hand the gates are much slower than on Skywater 130nm. Hopefully both programs will be around for a while, so if you are still interested when you have more time you can join the shuttle then.
  - hasheddan 2mo ago
    You can check out the tiles for this project and their exact location on the TT chip[0]. [0] https://tinytapeout.com/chips/ttgf26b/tt_um_coffeepot https://tinytapeout.com/chips/ttgf26b/tt_um_coffeepot
    - random__duck 2mo ago
      Nice, thanks for posting the link. For the people unfamiliar with the Tiny Tapeout program: everyone that gets a copy of the Tiny Tapeout chip will be getting a copy of all the ASICs on the shuttle.
roadbuster 2mo ago
Great project & article. Beyond the use of verilog, I couldn't find a description of the EDA stack. What tooling is used for the physical stages of design: place & route, LVS, DRC, and electrical circuit simulation (SPICE)?
- random__duck 2mo ago
  True, I went over the EDA stack a bit in a previous article here: https://talesonthewire.com/projects/blake2s_hashing_accelerator_a_solo_tapeout_journey/#open-source-silicon https://talesonthewire.com/projects/blake2s_hashing_accelera... This used the librelane classic flow, so it's a lot of verilator, yosys, openroad, opensta, magic and klayout. :)
brcmthrowaway 2mo ago
Was Claude used?
- random__duck 2mo ago
  No AI was used either for the project itself or for the post.
zrobotics 2mo ago
This is extremely cool, I've played with FPGA design but never really even looked into custom ASIC design due to budget concerns. I hadn't heard of the tinytapeout project. Any recommended links for getting started? I have a (really dumb, kinda meme-y) project involving a dead hakko soldering iron and "Internet of Soldering", a custom ASIC for the actual front-end would be the appropriate level of silly and possibly could fit in the project budget.
- random__duck 2mo ago
  As the designer of "an ethernet connected beacon that counts and never overflows until the heat death of the universe" (https://talesonthewire.com/projects/until_heat_death_do_us_part/ https://talesonthewire.com/projects/until_heat_death_do_us_p...) I believe you have found the perfect audience for a memeish hardware project. So don't expect anything but my upmost support. :) I detailed the roadmap for custom ASIC design a bit here: https://talesonthewire.com/projects/two_weeks_until_tapeout/#project-roadmap https://talesonthewire.com/projects/two_weeks_until_tapeout/... and joining the Tiny Tapeout discord community is a great place to reach out for help.
  - toast0 2mo ago
    > So for everyone else also running multiple instances of experimental hardware on their local network I have added an additional 2 Byte magic number using the all so original 0xCAFE (little endian) allowing packet parsers to distinguish between this ASIC’s experimental traffic and your other experimental traffic. Just saying, it's good form to do network stuff in big endian. Although since it's already done I dunno how hard you should work to retcon that to 0xFECA...
    - inigyou 2mo ago
      If your 0xCAFE gives you a 0xFECA1A55, stop going there...
      - toast0 2mo ago
jandrese 2mo ago
That memory constraint is a real killer on the chip, even one that is a first generation toy like this. Only having 4 entries in the forwarding table is going to be an immediate problem if that switch is ever connected to another switch. I wonder if it wouldn't make sense for one of the ports to be designated the "uplink" port and if the switch receives a packet destined for something not in the table it forwards it out the uplink port so long as the packet did not arrive via that port. It could even save a bit of memory by not storing entries for the uplink port in the table.
- Taniwha 2mo ago
  On TinyTapeout you can add more tiles up to 16 times the size of this (at which point you likely jump to waferspace)
- inigyou 2mo ago
  Correct ethernet switch behavior forwards a packet to all ports if the destination MAC is unknown
  - crote 2mo ago
    Sure, but at that point you're just building a complicated hub, and trying to actually operate it at speed would result in most packages getting dropped due to collisions as it is pretending to be full-duplex while actually only being half-duplex.
    - inigyou 2mo ago
      Ethernet switches are complicated hubs.
      - throw0101a 2mo ago
        Some of the 'old' names: > A network switch (also called switching hub, bridging hub, Ethernet switch, and—by the IEEE—MAC bridge[1]) is networking hardware that connects devices on a computer network by using packet switching to receive and forward data to the destination device. * https://en.wikipedia.org/wiki/Network_switch https://en.wikipedia.org/wiki/Network_switch
netik 2mo ago
How much of this is AI?
- random__duck 2mo ago
  No AI was used either for the project itself or for the post. I encourage you to look at the code and the git history.
effnorwood 2mo ago
Make it into a company. Name it Sanfran.
lukeh 2mo ago
You might want to make the port index instead a bit mask, that will let you support multicast and also selective broadcast flooding (although if you have limited space you’ll likely make broadcast flooding the default).

252 points by random__duck 2mo ago

“ Cut-through vs Store-and-Forward” I had a Google interview collapse because the last interviewer was a Principle Engineer who just couldn’t accept that a switch will ever forward a frame before reading in full and checking the FCS. I was too much of an idiot to go along with it and instead ended up having an argument with him.

random__duck 2mo ago
Sounds like they could have used you on that team. Bad luck, but it at least you can walk away proud knowing it was not an issue with your technical skills. :)
inigyou 2mo ago
Cut-through is applicable in a minority of scenarios which are sometimes common. The majority of switches, especially the ones that support any advanced features at all, can't use it. Even when you can use it, you still can't use it all the time because of port conflicts. As far as advanced features go, even adding and removing VLAN tags is a headache in a cut-through switch.
- userbinator 2mo ago
  As far as advanced features go, even adding and removing VLAN tags is a headache in a cut-through switch. Not really; it just needs to inject the tag right after the source address, or not pass it through at all; and of course recompute throughout and replace the FCS at the end. Other more advanced packet editing can be done in a streaming fashion too, with the latency only limited by the length of any "forward references" needed.
  - crote 2mo ago
    But "just recompute the FCS" isn't quite the right approach, is it? With a traditional cut-through switch the entire packet is passed as-is, so you're not modifying a corrupt package. You don't drop it, but the corruption is trivially detectable on the other side. If you naively recompute the FCS on transmit you essentially mark a corrupted package as valid, so you have to compute the FCS on transmit and receive, and intentionally send out the wrong FCS on transmit if you determine the receive FCS is invalid.
    - bcrl 2mo ago
      It's easy enough to write a bit of Verilog to perform partial updates on the CRC to only compensate for what has changed. I did so for an FPGA project I implemented a number of years ago when processing VLAN tags and doing IP forwarding. That said, there are lazy hardware designers that just slap on a block to recompute the CRC and replace the old one with the new. An experience I had years ago at Red Hat taught me that the hard way as a shiny new Cisco ethernet switch IT deployed ended up costing us weeks of frustration and delayed a release when it caused bit flips in NFS packets during stress testing. The switch was so awesomely advanced that it recomputed the CRC on the bit flipped packets causing kernel builds to fail. <sigh>
      - foobiekr 2mo ago
        Wow that rings a bell. The lack of end to end ECC protection in some packet paths on very high end ASICs is something I just can’t get over.
        bcrl 2mo ago
        ECC tends to be too expensive to be used widely in hardware data paths. Most of the time hardware will use a parity bit as checking it is far less costly (both in terms of gates and power). Parity bits are usually sufficient, as the purpose is really limited to identifying hardware failure. The CRC should prevent delivery of a corrupted packet provided the hardware doesn't cheat and blindly recompute it.
sophacles 2mo ago
I'm struggling to think of any real-world networking scenario where cut-through switching for ethernet can provide any sort of benefit. Most data-center networks are involve aggregation/spine/core switches at higher speed than TOR/leaf switches - so the packet has to be buffered. Most every workload is going to involve port collisions (2 packets at the same time needing the same output port) - again a full buffering required (or drops and throughput reduction). Bigger networks start using vlans, mpls, ip routing at switches, and vxlan - requiring more lookups per-packet, larger "minimum chunks of packet seen before port is determined" sizes, etc reducing the value of cut-through. And so on. But i also struggle to find a scenario where a principle engineer at google is refusing to accept that cut-through switches exist, but rather, was refusing to accept that the scenario they're talking through with you would ever involve an effectively deployed cut-through switch (since that's basically never).
- noncoml 2mo ago
  > but rather, was refusing to accept that the scenario they're talking through with you would ever involve an effectively deployed cut-through switch (since that's basically never). Nope, the argument was exactly if a cut-through switch exists or ever existed Edit: Actually the conversation came back to me: He asked me about the frame format. Once I drew that on the board, he asked why is the DST before the SRC. My answer was in order for the switch to start doing the lookups ASAP and that's when I mentioned that a cut-through switch can switch the packet before it even finishes receiving it. After that we spend the rest of the interview time arguing if cut-through switches actually exist or not. Both he and I having too big of an ego to let it go..
  - icedchai 2mo ago
    So he didn't want to google it?
    - stinkbeetle 2mo ago
      I see you've never had a run-in with a blowhard in a position of power. Lucky.
gjvc 2mo ago
"principal"
Aurornis 2mo ago
I remember studying cut-through forwarding in a class and thinking it was obviously going to be used everywhere. Then I was disappointed to enter the real world of networking and learn that it wasn’t actually used as often as I had been led to believe due to conflicting with higher priority features. So the interview may have been a conflict between textbook learning and the Google engineer’s practical experience. It’s tough to encounter that conflict in interviews because you get two people talking past each other.

inigyou 2mo ago

Nice! I've been following waferspace and tiny tapeout loosely but I'm sadly not in a position to be able to work on something right now. I'm surprised you managed to fit something like this in a TT grid square.

random__duck 2mo ago
You can join multiple Tiny Tapeout tiles (grid squares) together to form bigger projects, which is what I did here. It's actually one of the bigger 4 grid projects. Also how much you can fit really depends on the node the shuttle is targeting. So I was in luck that this was on Global Foundries 180nm as you can cram in a lot for flip-flops onto a single tile compared with Skywater 130nm. On the other hand the gates are much slower than on Skywater 130nm. Hopefully both programs will be around for a while, so if you are still interested when you have more time you can join the shuttle then.
- hasheddan 2mo ago
  You can check out the tiles for this project and their exact location on the TT chip[0]. [0] https://tinytapeout.com/chips/ttgf26b/tt_um_coffeepot https://tinytapeout.com/chips/ttgf26b/tt_um_coffeepot
  - random__duck 2mo ago
    Nice, thanks for posting the link. For the people unfamiliar with the Tiny Tapeout program: everyone that gets a copy of the Tiny Tapeout chip will be getting a copy of all the ASICs on the shuttle.

roadbuster 2mo ago

Great project & article. Beyond the use of verilog, I couldn't find a description of the EDA stack. What tooling is used for the physical stages of design: place & route, LVS, DRC, and electrical circuit simulation (SPICE)?

random__duck 2mo ago
True, I went over the EDA stack a bit in a previous article here: https://talesonthewire.com/projects/blake2s_hashing_accelerator_a_solo_tapeout_journey/#open-source-silicon https://talesonthewire.com/projects/blake2s_hashing_accelera... This used the librelane classic flow, so it's a lot of verilator, yosys, openroad, opensta, magic and klayout. :)

brcmthrowaway 2mo ago

Was Claude used?

random__duck 2mo ago
No AI was used either for the project itself or for the post.

zrobotics 2mo ago

This is extremely cool, I've played with FPGA design but never really even looked into custom ASIC design due to budget concerns. I hadn't heard of the tinytapeout project. Any recommended links for getting started? I have a (really dumb, kinda meme-y) project involving a dead hakko soldering iron and "Internet of Soldering", a custom ASIC for the actual front-end would be the appropriate level of silly and possibly could fit in the project budget.

random__duck 2mo ago
As the designer of "an ethernet connected beacon that counts and never overflows until the heat death of the universe" (https://talesonthewire.com/projects/until_heat_death_do_us_part/ https://talesonthewire.com/projects/until_heat_death_do_us_p...) I believe you have found the perfect audience for a memeish hardware project. So don't expect anything but my upmost support. :) I detailed the roadmap for custom ASIC design a bit here: https://talesonthewire.com/projects/two_weeks_until_tapeout/#project-roadmap https://talesonthewire.com/projects/two_weeks_until_tapeout/... and joining the Tiny Tapeout discord community is a great place to reach out for help.
- toast0 2mo ago
  > So for everyone else also running multiple instances of experimental hardware on their local network I have added an additional 2 Byte magic number using the all so original 0xCAFE (little endian) allowing packet parsers to distinguish between this ASIC’s experimental traffic and your other experimental traffic. Just saying, it's good form to do network stuff in big endian. Although since it's already done I dunno how hard you should work to retcon that to 0xFECA...
  - inigyou 2mo ago
    If your 0xCAFE gives you a 0xFECA1A55, stop going there...
    - toast0 2mo ago

jandrese 2mo ago

That memory constraint is a real killer on the chip, even one that is a first generation toy like this. Only having 4 entries in the forwarding table is going to be an immediate problem if that switch is ever connected to another switch. I wonder if it wouldn't make sense for one of the ports to be designated the "uplink" port and if the switch receives a packet destined for something not in the table it forwards it out the uplink port so long as the packet did not arrive via that port. It could even save a bit of memory by not storing entries for the uplink port in the table.

Taniwha 2mo ago
On TinyTapeout you can add more tiles up to 16 times the size of this (at which point you likely jump to waferspace)
inigyou 2mo ago
Correct ethernet switch behavior forwards a packet to all ports if the destination MAC is unknown
- crote 2mo ago
  Sure, but at that point you're just building a complicated hub, and trying to actually operate it at speed would result in most packages getting dropped due to collisions as it is pretending to be full-duplex while actually only being half-duplex.
  - inigyou 2mo ago
    Ethernet switches are complicated hubs.
    - throw0101a 2mo ago
      Some of the 'old' names: > A network switch (also called switching hub, bridging hub, Ethernet switch, and—by the IEEE—MAC bridge[1]) is networking hardware that connects devices on a computer network by using packet switching to receive and forward data to the destination device. * https://en.wikipedia.org/wiki/Network_switch https://en.wikipedia.org/wiki/Network_switch

netik 2mo ago

How much of this is AI?

random__duck 2mo ago
No AI was used either for the project itself or for the post. I encourage you to look at the code and the git history.

effnorwood 2mo ago

Make it into a company. Name it Sanfran.

lukeh 2mo ago

You might want to make the port index instead a bit mask, that will let you support multicast and also selective broadcast flooding (although if you have limited space you’ll likely make broadcast flooding the default).